Title: Maximum Average Randomly Sampled: A Scale Free and Non-parametric Algorithm for Stochastic Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: Stochastic Multi-armed bandit

Online Learning

Upper Confidence Bound
Abstract: Upper Confidence Bound (UCB) methods are one of the most effective methods in dealing with the exploration-exploitation trade-off in online decision-making problems. The confidence bounds utilized in UCB methods tend to be constructed based on concentration equalities which are usually dependent on a parameter of scale (e.g. a bound on the payoffs, a variance, or a subgaussian parameter) that must be known in advance. The necessity of knowing a scale parameter a priori and the fact that the confidence bounds only use the tail information can deteriorate the performance of the UCB methods.

Here we propose a data-dependent UCB algorithm called MARS (Maximum Average Randomly Sampled) in a non-parametric setup for multi-armed bandits with symmetric rewards. The algorithm does not depend on any scaling, and the data-dependent upper confidence bound is constructed based on the maximum average of randomly sampled rewards inspired by the work of Hartigan in the 1960s and 70s. A regret bound for the multi-armed bandit problem is derived under the same assumptions as for the $\psi$-UCB method without incorporating any correction factors. The method is illustrated and compared with baseline algorithms in numerical experiments.

Maximum Average Randomly Sampled: A Scale Free and Non-parametric Algorithm for Stochastic Bandits

Abstract

Upper Confidence Bound (UCB) methods are one of the most effective methods in dealing with the exploration-exploitation trade-off in online decision-making problems. The confidence bounds utilized in UCB methods tend to be constructed based on concentration equalities which are usually dependent on a parameter of scale (e.g. a bound on the payoffs, a variance, or a subgaussian parameter) that must be known in advance. The necessity of knowing a scale parameter a priori and the fact that the confidence bounds only use the tail information can deteriorate the performance of the UCB methods. Here we propose a data-dependent UCB algorithm called MARS ( M aximum A verage of R andomly S ampled Rewards) in a non-parametric setup for multi-armed bandits with symmetric rewards. The algorithm does not depend on any scaling, and the data-dependent upper confidence bound is constructed based on the maximum average of randomly sampled rewards inspired by the work of Hartigan in the 1960s and 70s. A regret bound for the multiarmed bandit problem is derived under the same assumptions as for the ψ -UCB method without incorporating any correction factors. The method is illustrated and compared with baseline algorithms in numerical experiments.

1 Introduction

The classical stochastic multi-armed bandit problem introduced by [21] has been studied extensively in statistics, computer science, electrical engineering, and economics. New applications in machine learning and data science such as personalized new recommendation and Monte-Carlo tree search have generated renewed interest in the problem.

The problem is described as follows: The classical stochastic multi-armed bandit problem is an online decision-making problem where each arm represents an action. At each round t ∈ { 1 , · · · , n } the agent chooses an action (arm) A t ∈ A and observe a reward X t ∈ R where A is the set of available actions and its finite cardinality is K ≥ 2 . The reward X t is a sample from an underlying distribution

ν A t with bounded mean µ A t . Hence, a stochastic bandit is a set of distributions ν = { ν i : i ∈ A} which is unknown for the agent.

The agent aims to minimize the following regret over n rounds.

with respect to the actions A 1 , . . . , A n .

The minimization of the regret necessitates balancing the trade-off between exploration and exploitation where exploration means investigation to accumulate more information about arms and exploitation means the maximization of the immediate performance. See the book [19] and survey [7] for comprehensive discussion.

A class of algorithms for multi-armed bandit problems that strikes a balance between exploration and exploitation is Upper Confidence Bound (UCB) methods [3, 4, 12, 1].

The UCB algorithm is based on the principle of optimism in the face of uncertainty . According to this principle, one should construct probabilistically optimistic guesses for the expected payoff (mean) of all actions and play the arm with the highest guess. The performance of the UCB algorithms in dealing with the exploration-exploitation dilemma depends on the tightness of the upper confidence bound. The literature includes several approaches which construct upper confidence bound based on concentration inequalities such as Hoeffding-type inequalities [4] and self-normalized inequalities [1].

As an example ψ -UCB Method introduced in [7] assumes that there exist a convex function ψ ( · ) which satisfy the following conditions.

Let ψ ∗ ( · ) be the Legendre-Fenchel transform of ψ ( · ) , that is

Then, with probability at least 1 -δ ,

where ˆ µ i,s is the sample mean of rewards obtained by pulling arm i for s times.

At time t , using the obtained UCB and letting δ = 1 /t 2 , we select

where T i ( t -1) denotes the number of times action i was chosen by the learner after the end of round t -1 . It was proved [7] that:

where ∆ i is the suboptimality gap ∆ i = max i ∈A µ i -µ i .

However, there are two weaknesses associated with the bounds obtained by concentration inequalities as above. First, they are commonly conservative since they are not data-dependent and only care about tail information of distribution. Second, those bounds usually depend on scaling factors or functions, e.g. ψ ( · ) in ψ ( · ) -UCB and poor choices can make the bounds conservative.

1.1 Related Works

Recently several works have focused on non-parametric bandit algorithms based on subsampling and bootstrapping [5, 6, 18, 17, 22]. Those works use the empirical distribution of the data instead of fitting a given model to the data. The Garbage In, Reward Out (GIRO) method [18] relies on the

history of past observed rewards and enhances its regret bound by augmenting fake samples into the history. The Perturbed-History Exploration method (PHE) [17] serves as a faster and memory-efficient alternative to GIRO. However, PHE has the limitation of being restricted to bounded distributions and involves a tunable parameter. The Residual Bootstrap exploration (ReBoot) [22] perturbs the history in order to improve the regret bound.

Bootstrapping Upper Confidence Bound uses bootstrap to construct sharper confidence intervals in UCB-type algorithm [12]. However, a second-order correction is used to guarantee the nonasymptotic validity of the bootstrap threshold. The second-order correction is not sharp, and it includes scaling factors.

Another line of works which include the Best Empirical Sampled Average method (BESA) [5] and the Sub-sampling Duelling Algorithms (SDA) [6] use subsampling to conduct pairwise comparison (duels) between arms. BESA organize the duels between arms and find the winner by comparing the empirical average of sub-sampled rewards. SDA extends the concept of BESA duels to a round-based comparison by incorporating a sub-sampling scheme and it eliminates the need for forced exploration.

Apart from Reboot which was analysed for Gaussian distributions, and SDA which was analysed for a family of distribution satisfying a balance condition (including Gaussian and Poisson), the other algorithms were analysed for distributions with known bounded support.

1.2 Contributions

In this paper, we propose a new non-parametric UCB algorithm which is fully data-dependent without any scale parameter. The proposed approach called MARS ( M aximum A verage R andomly S ampled) constructs a data-dependent upper confidence bound by selecting the maximum average among randomly sampled rewards. This upper confidence bound is inspired by Leave-out Signdominant Correlation Regions (LSCR) method which construct non-asymptotic confidence regions for parameters of dynamical systems [8, 9, 11, 15, 20, 16]. The upper confidence bound inspired by LSCR is non-asymptotic and probabilistically guaranteed , i.e. it is larger than the true value with an exact user-chosen probability and for a finite number of data points.

The problem-dependent regret bound of the method is given for multi-armed bandits where all underlying distributions { ν i , i ∈ A} are symmetrically distributed about their mean e.g. exponential, uniform, or logistic distribution and inequality (2) is satisfied. As ψ ( · ) does not need to be known, MARS avoids the problem with sub-optimal choice of ψ ( · ) , and still achieves logarithmic regret.

1.3 Notation

A subset of positive integers up to and including a constant k is denoted as [ k ] ≜ { 1 , · · · , k } . ⌈ x ⌉ is the smallest integer greater than or equal to x . The cardinality of a set S is denoted by | S | , e.g. | [ k ] | = k . I { A predicate } is the indicator function defined as

Without loss of generality assume the first arm is optimal i.e. µ 1 = max i µ i . Define the sub-optimality gap ∆ i = µ 1 -µ i for i ∈ { 2 , · · · , K } . Further, let

be the number of times action i was chosen by the learner after the end of round t . X i,j is the reward the j th time arm i is pulled. U ( a, b ) is a uniform distribution bounded by a and b ( a < b ), and B ( a ) is a Bernoulli distribution with mean a .

The remainder of the paper is organized as follows. In the next section, a new randomized approach to compute an upper confidence bound is presented. In Section 3 the non-parametric UCB algorithm is introduced and a regret bound for the multi-armed bandit problem with symmetric rewards is given. Section 4 explores the numerical performance of the proposed MARS method, and Section 5 concludes the paper.

2 A New Randomized Upper Confidence Bound

In this section, an approach to compute a data-dependent upper confidence bound for a finite number of observations is proposed.

Hartigan in [14] proposed an approach which is useful to compute confidence interval for an unknown parameter. The confidence interval was computed by a number of estimates of the mean using different subsets of the data. The estimates of the mean in [12] was computed using a number of balanced sets of subsamples determined by group theory.

The paper [9] used random strings instead of balanced sets of subsamples which brings computational advantage. Now, we use the method in [14] and [9] to propose a new non-asymptotic upper confidence bound.

We have the observations { X i,s } T i s =1 from arm i and we want to find an upper confidence bound UCB i with probability 1 -δ such that

In this section T i is assumed to be a fixed number of observations which does not depend on other actions taken or observed rewards. In Section 3, the situation where T i is dependent on the past actions and observed rewards will be considered.

Before introducing the new upper confidence bound, let us define typical values.

Definition 1. The set of ascending-ordered and continuously distributed random variables Z (1) , Z (2) , . . . , Z ( M -1) is a set of typical values for µ if the probabilities that µ belongs to each of following intervals are the same

that is

Now, based on Theorem 1 in [9] the following theorem introduces a set of typical values for the means of i th arm µ i .

Theorem 1. We have the observations { X i,s } T i s =1 from arm i and E { X i,s } = µ i for all s ∈ [ T i ] . Assume that the observations { X i,s } admit continuous distributions and are independent and symmetrically distributed about their mean.

Then, a set of typical values for µ i is given by

where M ≤ 2 T i is a positive integer and { h j,t } are independent random sequences such that

A string { h j,s } N s =1 is discarded if it turns out to be equal to an already constructed string.

Proof. Follows along the same line as the Proof of Theorem 1 in [9]. See Section 1 in the supplement for further details.

Theorem 1 shows that M -1 estimates of mean computed by random sub-sampling are a set of typical values for µ i and partition the real line into equiprobable segments where µ i belong to each one of those segments with equal probability. As a consequence of Theorem 1 we conclude that

M is a user-chosen parameter and hence the probability of the confidence bound can be tuned by the user. Since M ≤ 2 T i the maximum confidence which can be achieved by computing subsample

means is 1 -1 / 2 T i and we can not get an upper confidence bound with arbitrary high confidence. To overcome this problem, we use the fact that all means µ i are bounded and the following upper confidence bound is proposed.

Compute M = ⌈ δ -1 ⌉ random subsample means as in (4). The non-parametric upper confidence bound is given in Table 1.

The probability of the constructed upper confidence bound is presented in the next theorem.

Theorem 2. Let T i be a fixed number of observations from arm i . For each i ∈ [ K ] assume { X i,s } T i s =1 are independent random variables with continuous distribution functions which are symmetrically distributed about its mean µ i . Let the user-chosen parameter δ ∈ (0 , 1) be chosen such that δ -1 is an integer. Then for all T i > 0 the following non-asymptotic equality holds

Note that T i is deterministic, and the probability is with respect to both { X i,s } and { h j,s } .

Proof. See Section 2 in the supplement.

Theorem 2 shows that the probability of the proposed confidence region is exact and hence it is not conservative. This result holds for a finite number of observations as long as the reward is symmetrically distributed around its mean and T i is a fixed given number.

Example 1 In Figure 1, we compare different approaches to calculate 95% UCB for the population mean based on samples from a Gaussian and an uniform distribution. The upper confidence bounds are calculated for different number of data points. The almost exact quantile is computed by finding 95% quantile of the population of sample means available by repeating experiment 10000 times. Naive bootstrap threshold and bootstrapped threshold were computed by equation (2.6) in Remark 2.3 in [12].

The figures shows that for most sample sizes the MARS bound is sharper than naive bootstrapped threshold, bootstrapped threshold, sub-Gaussian bound, and Hoeffding bound. The naive bootstrap is sharper for small sample sizes, but as shown in [12], naive bootstrap do not produce reliable results for small samples sizes. It is also notable that the MARS bound is probabilistically guaranteed for a finite number of data points as shown in Theorem 2.

3 Algorithm and Regret Bound

The UCB method using the proposed upper confidence bound for n rounds is given in Algorithm 1. At the initial rounds, the algorithm tends to adopt an optimistic outlook in the face of uncertainty. This is reflected in the upper confidence bound, which is set to infinity or the highest reward observed. As the number of rounds increases, the upper confidence bounds approach the mean reward for each arm with high probability. Consequently, the algorithm progressively leans towards exploitation.

If two or more UCBs in line 15 of Algorithm 1 take on the same value, randomly choose an action among the minimizers.

MARS necessitates keeping ⌈ 1 /δ ⌉ sub-sampled means for each arm. This leads to a memory requirement of O ( Kn ) when δ = 1 /n . The computational complexity of MARS is also depends

0

100

200

300

400

500

Number of Data

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Upper Confidence Bound

exact quantile

MARS bound

naive Bootstrapped threshold

Bootstrapped threshold

Sub-Gaussian bound

Hoefeding bound

(a)

0

100

200

300

400

500

Number of Data

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

Upper Confidence Bound

exact quantile

MARS bound

naive Bootstrapped threshold

Bootstrapped threshold

Sub-Gaussian bound

(b)

Algorithm 1 M aximum A verage of R andomly S ampled (MARS)

on the choice of δ , as updating ⌈ 1 /δ ⌉ sub-sampled means is performed in each round. Notably, to reduce the computational burden, the required Bernoulli variables in the algorithm can be generated and stored before the start of the game.

The Next theorem provides a regret bound for the proposed MARS algorithm when applied to multi-armed bandit problems with symmetric distributions.

Theorem 3. Let T i ( n ) > 0 be the total number of observation from arm i and δ = 1 /n 2 . The following assumptions about all arms i ∈ [ K ] are in place.

{ X i,s } T i ( n ) s =1 are continuously distributed and independent random variables symmetrically distributed about its mean µ i . The observations X i,s for each arm i ∈ [ K ] are independent of the observations from other arms and actions.

For all i ∈ [ K ] and s ∈ [ T i ( n )] , there is a convex function ψ i ( λ ) such that:

Then the regret is bounded by where

and ψ ∗ i ( · ) is the Legendre-Fenchel transform of ψ i ( · ) , that is

Note that the expectation in the regret is with respect to both { X i,s } and { h j,s } .

Proof. See Section 3 in the supplement.

In Theorem 3 we set δ = 1 /n 2 to simplify analysis. A similar regret bound can also be achieved using δ t = 1 /t 2 . It is also notable that the regret bound in Theorem 3 was achieved without any knowledge about the functions ψ i ( λ ) apart from its existence. In particular, it was not used when the upper confidence bounds were computed. However, the ψ -UCB method assumed that ψ i ( λ ) s are known and used it in constructing the upper confidence bound.

The next corollary characterises a relationship between the regret bounds of the current method and those of the ψ -UCB method.

Corollary 1. Provided that 0 < ψ ∗ i (∆ i ) < 1 . 59 , the regret bound (6) is simplified as follows

Now, let for all arms i ∈ [ K ] , the corresponding reward is σ i -sub Gaussian. Then, ψ ∗ i (∆ i ) = (∆ 2 i ) / (2 σ 2 i ) . Accordingly when (∆ 2 i ) / (2 σ 2 i ) < 1 . 59 the regret bound is simplified to

Proof. See Section 4 in the supplement.

As shown in equation (6), the regret bound for MARS is always O (log( n )) without relying on the use of ψ ( · ) . This demonstrates the effectiveness of the introduced non-parametric UCB method. When ψ ∗ i (∆ i ) < 1 . 59 , the task becomes more challenging as identifying the optimal arms becomes harder. In such scenarios, both the regret bounds for the proposed MARS and the ψ -UCB, which employs ψ ( · ) , become dependent on the function log( n ) /ψ ∗ i (∆ i ) . Corollary 1 also explore the effectiveness of MARS when dealing with subgaussian rewards. It demonstrates that even without prior knowledge of the σ i values, MARS successfully addresses bandit problems, achieving a regret bound of O ( ∑ i :∆ i > 0 log( n ) / ∆ i ) for challenging scenarios where (∆ 2 i ) / (2 σ 2 i ) < 1 . 59 .

4 Experiments

In this section, the performance of the proposed MARS method is compared with upper confidence bound based on concentration inequalities (Vanilla UCB, See e.g. Chapter 7 and 8 in [19]), Thompson sampling with normal prior (Normal-TS) [2], Bootstrapped UCB [12], Garbage In, Reward Out (GIRO) [18], and Perturbed-History Exploration (PHE) [17]. The package PymaBandits 2 was used to implement these baseline methods.

The probability of confidence regions in the Vanilla UCB, Bootstrapped UCB, and the proposed MARS UCB is set to 0 . 999 , i.e. δ = 1 / 1000 . In GIRO, the parameter a , which represents pseudorewards per unit of history, is set to 1 . Due to the high sensitivity of the PHE approach to the choice of the tunable parameter a , simulations were performed for two values of a .

The number of arms is K = 5 and the means are

First, consider the case where the rewards are Gaussian with variance 1 for all arms. The cumulative regrets are show in Figure 2a. Since Normal-TS uses distribution knowledge and the variances are correct it is the best as expected. The Vanilla UCB algorithm demonstrates comparable or superior performance compared to both the Bootstrapped UCB, MARS, and GIRO. The performance of the PHE approach is heavily dependent on the parameter a . When a = 2 . 1 , it shows a linear regret. However, for a = 5 . 1 , it outperforms most other approaches, except for Thompson sampling.

Both Vanilla UCB and Normal-TS depend on the variances which were assumed known in the previous simulation. We repeat the simulation with the variances incorrectly set to 2. The result is shown in Figure 2b. Evidently MARS, GIRO, and Bootstrapped UCB outperform both Vanilla UCB and Normal-TS when incorrect values for the variances are used. MARS demonstrates superior performance over Bootstrapped UCB and GIRO after an initial set of rounds. Moreover, unlike GIRO and Bootstrapped UCB, MARS does not require the full storage of reward history, resulting in lower computational complexity. As previously observed, PHE with a value of 2 . 1 demonstrated the poorest performance, whereas PHE with a value of 5 . 1 has the best performance.

0

250

500

750

1000

1250

1500

1750

2000

Round

20

40

60

80

Regret

Vanila-UCB

Normal-TS

Bootstrapped UCB

MARS

GIRO

PHE (a=2.1)

PHE (a=5.1)

(a)

0

250

500

750

1000

1250

1500

1750

2000

Round

0

20

40

60

80

100

120

Regret

Vanila-UCB

Normal-TS

Bootstrapped UCB

MARS

GIRO

PHE (a=2.1)

PHE (a=5.1)

(b)

The MARS and GIRO do not use the tail information of the rewards. However, Vanilla UCB, NormalTS, and Bootstrapped UCB use the distribution and the tail information of the rewards respectively and their performance can deteriorate when the prior knowledge is wrong or conservative. To illustrate this we repeated the simulation for the cases where the rewards admit uniform distribution over [ -1 , 1] . The results are shown in Figure 3. It shows that MARS which does not use the distribution of the rewards and the tail information, outperforms the other methods except PHE ( a = 2 . 1 ) in this case. An interesting observation is that PHE ( a = 5 . 1 ) exhibits outstanding performance in a Gaussian setup, yet it performs poorly in a Uniform setup, indicating a strong reliance on the tunable

parameter. This dependence on the parameter could pose challenges in real-world applications where the environment is unknown. In such cases methods like MARS and GIRO may be more practical alternatives.

0

250

500

750

1000

1250

1500

1750

2000

10

20

30

40

50

60

70

Regret

UCBVan

Thompson

BootstrapUCB

MARS

GIRO

PHE (a=2.1)

PHE (a=5.1)

Round

See Section 5 in the supplement for further simulations.

5 Conclusion

In this paper, a non-parametric and scale-free confidence bound was proposed based on random sub-sampling of rewards. The confidence bound was used to propose a data-driven UCB algorithm. A regret bound of the method for multi-armed bandit problems with symmetric distribution was presented, and it was shown that the regret bound is always O (log( n )) . Moreover, the regret bound is O ( ∑ i :∆ i > 0 log( n ) / ∆ i ) for a class of Bandit problems. Numerical experiments show that the proposed method performs well, and it outperforms baselines algorithms when the prior information is mis-specified.

Extending the method to other bandit and reinforcement learning problems are interesting topics for future works e.g. evaluating the method on non-stationary bandits and contextual bandits using the ideas in [13] and [9]. As MARS does not use any concentration inequalities in the construction of the confidence bound, the application of MARS to bandits with heavy tail is another interesting direction. Another interesting future direction is evaluation of the method on asymmetric rewads and robustification of the approach following ideas in [10].

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems , pages 2312-2320, 2011.

Shipra Agrawal and Navin Goyal. Further optimal regret bounds for thompson sampling. In Artificial intelligence and statistics , pages 99-107, 2013.

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2-3):235-256, 2002.

Akram Baransi, Odalric-Ambrym Maillard, and Shie Mannor. Sub-sampling for multi-armed bandits. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 115-131. Springer, 2014.

Dorian Baudry, Emilie Kaufmann, and Odalric-Ambrym Maillard. Sub-sampling for efficient nonparametric bandit exploration. Advances in Neural Information Processing Systems , 33:5468-5478, 2020.

Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning , 5(1):1-122, 2012.

Marco C. Campi and Erik Weyer. Guaranteed non-asymptotic confidence regions in system identification. Automatica , 41(10):1751-1764, 2005.

Marco C. Campi and Erik Weyer. Non-asymptotic confidence sets for the parameters of linear transfer functions. IEEE Transactions on Automatic Control , 55(12):2708-2720, 2010.

Algo Care, Balázs Csanád Csáji, and Marco C. Campi. Sign-perturbed sums (SPS) with asymmetric noise: Robustness analysis and robustification techniques. In 2016 IEEE 55th Conference on Decision and Control (CDC) . IEEE, 2016.

Marco Dalai, Erik Weyer, and Marco C. Campi. Parameter identification for nonlinear systems: Guaranteed confidence regions through LSCR. Automatica , 43(8):1418-1425, 2007.

Botao Hao, Yasin Abbasi Yadkori, Zheng Wen, and Guang Cheng. Bootstrapping upper confidence bound. In Advances in Neural Information Processing Systems , pages 12123-12133, 2019.

JA Hartigan. Exact confidence intervals in regression problems with independent symmetric errors. The Annals of Mathematical Statistics , pages 1992-1998, 1970.

John Hartigan. Using subsample values as typical values. Journal of the American Statistical Association , 64(328):1303-1317, 1969.

Masoud Moravej Khorasani and Erik Weyer. Non-asymptotic confidence regions for the parameters of EIV systems. Automatica , 115:108873, 2020.

Masoud Moravej Khorasani and Erik Weyer. Non-asymptotic confidence regions for the transfer functions of errors-in-variables systems. IEEE Transactions on Automatic Control , 67(5):2373-2388, 2021.

Branislav Kveton, Csaba Szepesvari, Mohammad Ghavamzadeh, and Craig Boutilier. Perturbed-history exploration in stochastic multi-armed bandits. arXiv preprint arXiv:1902.10089 , 2019.

Branislav Kveton, Csaba Szepesvari, Sharan Vaswani, Zheng Wen, Tor Lattimore, and Mohammad Ghavamzadeh. Garbage in, reward out: Bootstrapping exploration in multi-armed bandits. In International Conference on Machine Learning , pages 3601-3610. PMLR, 2019.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms. preprint , page 28, 2018.

Masoud Moravej Khorasani and Erik Weyer. Non-asymptotic confidence regions for errors-in-variables systems. In 18th IFAC symposium on system identification , volume 51, pages 1020-1025, Stockholm, Sweden, July 2018. IFAC-PapersOnLine.

Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society , 58(5):527-535, 1952.

Chi-Hua Wang, Yang Yu, Botao Hao, and Guang Cheng. Residual bootstrap exploration for bandit algorithms. arXiv preprint arXiv:2002.08436 , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel non-parametric algorithm for stochastic bandits, based on a sub-sampling scheme. While other algorithms using sub-sampling have been recently proposed, their approach significantly differs from these works. Indeed, instead of using sub-sampling to perform pairwise comparisons, they use it to build exact confidence intervals used in a UCB algorithm called MARS. To do that, they build on results provided by statisticians in the 60s and 70s, showing that sub-sampling can be used to build a set of *typical values* for the true mean of each distribution.\n\nAfter describing their method, the authors provide the theoretical guarantees of MARS: logarithmic regret under the assumption that the distributions are continuous and symmetric, and some exponential moment condition similar to the one used for $\\psi$-UCB. Interestingly, these guarantees are achieved while the algorithm does not use the function $\\phi$ directly, hence avoiding sub-optimal choices of $\\psi$, which is the advantage of non-parametric algorithms.\n\nThen, a set of experiments aim at validating the method.\n\nSTRENGTHS:\nI think that the paper is overall well-written and clear, and that the algorithm and theoretical results are well presented and easy to understand. \n\nWhile the literature in bandits is now quite vast, it is difficult to come up with new principles, especially in the standard stochastic case. Recently, the literature on non-parametric algorithms based on bootstrap and sub-sampling has proposed interesting new approaches, and this paper is in my opinion a nice addition to this literature, showing new potentialities. In particular, the theoretical guarantees hold under original assumptions, for which no other algorithms is proved to work (additional assumptions would be needed). Furthermore, the empirical results validate the approach.\n\nOverall, I think that the paper brings enough interesting insights for publication at Neurips, up to some changes listed below. The derivation of the technical results seem correct to me.\n\nWEAKNESSES:\nIn my opinion the paper has two major weaknesses, that are related:\n\n* I believe that in the context of the paper the literature review on non-parametric algorithms should be more precise, and the contribution of MARS compared to these approaches should be better explained. First, the presented algorithms are not all based on \"subsampling\", GIRO, PHE, Bootstrap-UCB and Reboot are closer to bootstrap, while only BESA and SDA are technically speaking based on sub-sampling. Those approaches are rather different, work under different assumptions, and do not share the same pros and cons. \n\nFurthermore, the last line \"One drawback of...\" is misleading, and make the reader believe that the said drawbacks are shared by all these algorithms, would be solved by MARS, and are in fact the motivation for MARS; while it seems to me that this all these perceived messages are wrong. \n\nIn my opinion, there are several axis that should be detailed when talking about these algorithms: the family of distributions for which they work (and maybe how tight are their guarantees for these families), what they need to know about the distributions, and their cost (memory and time). For instance, PHE is a faster and memory-less alternative to GIRO, but its guarantees are sub-optimal and restricted to bounded distributions, and there is a tunable parameter (but this method easily generalize to structured settings, which is its main strength). On the other hand, SDA works in much broader setting (families of distributions for which a \"balance\" condition is satisfied), but this generality costs the fact of storing all observations in memory: this is in my opinion similar to what is obtained with MARS. Furthermore, the question of improving the computation time and memory for SDA has been studied for the LB-SDA algorithm (http://proceedings.mlr.press/v139/baudry21b/baudry21b.pdf). \n\n* (Related) the memory and computation time of MARS are not well discussed. In particular, it seems worse than most existing approaches: maintaining the sub-sample means for each arm costs $O(Kn^2)$ (if $\\delta=1/n^2$ is set), while at most $O(n)$ is needed for the benchmarks. Furthermore, the update of the UCB also requires to sample $n^2$ Bernoulli variables at each time step, which is also much larger the cost of (even the less \"optimized\") existing non-parametric approach. Those are major drawbacks in my opinion, that should be explicitly discussed, providing comparisons with benchmarks. However, this does not mean than the approach is not interesting: I believe that the authors should really focus on the fact that their algorithms have guarantees under original assumptions for which there are no competitors (or no tight competitors).\n\n* The contribution of the paper is not on the technical side, most results and techniques used by the authors are known, but I think that it is interesting enough so that it is not an issue.\n\n* I find the set of experiments not very convincing: only a few curves/benchmark algorithms are used, and I don't understand why $\\delta$ is not set to the theoretically valid threshold ($4\\times 10^{-6}$). In my opinion SDA (for instance LB-SDA, which is the fastest SDA) should be present in the benchmarks since it is theoretically valid for all the settings considered. I would like to see a comparison of computation times too.\n\nI believe the authors left enough space so that all these discussions could be added. In my opinion they should be carefully addressed in the revision, and I would be happy to further raise my score in that case.\n\nQUESTIONS:\n* The statistical validity of the estimator holds for large enough sample size. For smaller sample sizes, the author build a UCB using the *maximum of observed data and a fixed probability of returning $\\infty$ to keep the $\\delta$ confidence. At first read, this step is a bit confusing, and It may have been easier to simply provide the required $\\log(n)/\\log(2)$ samples per arm at the initialization of the algorithm. Is there an empirical motivation for not doing that (because from the theoretical point of view this do not seem to change much)?  \n* Related question: is it clear that the $2^{-T_i(n)}$ is tight? Since it is critical in terms of computation/memory cost it may be interesting to optimize it.\n* It would be interesting to understand how good the theoretical results are. You started to do that by comparing them with $\\psi$-UCB and the usual bound with $\\Delta^2$. Did you try to look at the Burnetas & Katehakis lower bound for this specific family of distributions?\n* Did you investigate an anytime version of MARS? It seems to me that the proof may easily adapt, making the algorithm more flexible.\n\nLIMITATIONS:\nNo potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new approach to develop a data-dependent upper confidence bound to replace the classical UCB based on concentration inequalities. The data-dependent bound is constructed using sub-sampling of rewards and offers a tighter estimate on the error than the classical UCB resulting in improved performance.\n\nSTRENGTHS:\nThe approach adopted in the paper is quite interesting and seems to be significantly simpler than existing studies on data-dependent bounds. Although Theorem 1 is based on an existing result, its application to bandits is interesting and I haven't seen that before.\n\nWEAKNESSES:\nI am convinced that there is some novelty in the paper, especially on the theoretical aspect. However, I feel that it may not be sufficient for a paper like this to warrant a publication. While data-dependent bounds are interesting to study and analyze, they themselves have little to offer in terms of advancing the theoretical understanding of multi-armed bandits. These collection of studies that focus on data-dependent bounds are, at a fundamental level, studies that focus on improving practical implementation of MAB. Specifically, as pointed out even by the authors, the classical UCB requires some additional information, that is often difficult to obtain in practice, thereby making such data-dependent approaches a practical alternative.\n\nFor a result that is fundamentally based on improving practical implementation, the paper has very limited empirical evaluation. I strongly suggest that the authors add more detailed empirical evaluation, both in terms of comparison with more algorithms like GIRO and PHE and in terms of more complicated examples, preferably a real world example. \n\nA more extensive empirical evaluation will demonstrate the actual improvement offered in practice by this new data-dependent estimator and the existing theoretical results will _support_ and _explain_ those results, making a strong overall paper.\n\nI am willing to increase my score if the authors can add more detailed experiments with proper analysis. I do understand that the rebuttal period might not be sufficient for that in which case, I suggest the authors to resubmit with more experiments. The results definitely seem promising. \n\nEDIT: Score updated based on rebuttal.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a new method to compute the upper confidence bound (UCB). The new method does not require knowing or estimating the scale parameters. The property shown by Theorem 2 addresses the conservative issue of the existing methods. The practical importance of free from modeling the scale parameters is verified in the experiments.\n\nSTRENGTHS:\n- (a) The paper contributes a new way to compute UCBs of practical importance; it does not require scale information.\n- (b) Rigorous theoretical analysis.\n- (c) Experiments demonstrate the advantage of the proposed UCBs.\n\nWEAKNESSES:\n- (d) What are typical values? It is defined in Definition 1, but its intuition and usage are unclear. Also, it would be great to explain the typical value's role in proving Theorem 2 (the only mention I saw of it is in the proof of Proposition 2).\n- (e) Maybe adding a discussion (after Line 192) about recovering the worst-case bound would make the paper more complete?\n- (f) Is it better to start the paper with an Introduction?\n\nQUESTIONS:\n- (g) The symmetric assumption restricts the applicability of this submission. Can we remove it? What would the challenge(s) be?\n- (h) Visualizing the differences between the choice of scale parameters (Figure 1) and the choice of prior (Figure 2) is nice. Is it possible to demonstrate the difference between the conservative approaches (Lines 51--54) and the non-conservative (Lines 141--143) UCBs?\n- (i) Between the scale-free (this submission) and the scale-aware (baselines compared in this submission) approaches, another approach would be estimating the unknown scale parameters. If some scale-estimation approach does not require the symmetric assumption, it might be a strong baseline to be compared. A related paper would be (http://proceedings.mlr.press/v119/zhu20d/zhu20d.pdf).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn multi-armed bandits (MAB), when the noise distribution is known, the UCB algorithm with a carefully constructed confidence bound achieves a gap-dependent regret depending on the noise distribution. This manuscript studies the following question: when the noise distribution is unknown, is there an algorithm that adaptively constructs the confidence bound and achieves a near-optimal regret? To answer this question, the authors used a cute observation in [Campi and Weyer, 2010]: if the distribution of a random variable is symmetric around its mean, then the sample means after subsampling (with probability 1/2) splits the real line into several equiprobable pieces for the true mean. This provides a fully data-driven approach to construct the upper confidence bound, and this manuscript shows that the resulting algorithm nearly achieves the optimal regret when the noise distribution is known. Experiments are also provided.\n\nSTRENGTHS:\nOverall I like this paper. This paper asks a clean question and provides a satisfactory answer to it. Although the main observation comes from [Campi and Weyer, 2010], and the analysis of the algorithm essentially parallels the traditional UCB analysis, in my opinion the application to bandit problems is still nice and makes this paper interesting enough for a NeurIPS publication.\n\nWEAKNESSES:\nI don't see a huge weakness of this work, but I can list some minor ones: \n\n1. The main novelty is the application of the observation in [Campi and Weyer, 2010] to a data-driven confidence bound in bandits, and everything else in the paper is pretty standard. \n\n2. The requirement that the distribution is symmetric around its mean is pretty restrictive. \n\n3. The computational complexity of the resulting algorithm could be high and may not lead to a practically useful algorithm. \n\nDespite the above concerns, I'd like to reiterate that the merits weigh more than the weaknesses in my opinion, and I still like this paper. \n\nDetailed comments: \n\nTable I: I think the probabilities on the first two rows should be swapped. \n\nSection 1: In my opinion, Section 1 should be a standalone section without resorting to bandits. In particular, the subscripts i should be removed. \n\nTheorem 2, or the case 2^T < M: is the modification in Table I really necessary for the application to MAB? After inspecting the proof, I feel that simply choosing UCB(T, delta) = \\infty whenever T < log_2(1/delta) also gives the same regret upper bound; is that true?\n\nQUESTIONS:\n1. Could the authors say anything about adaptivity (possibility or impossibility) when the reward distribution is not symmetric around the mean? \n\n2. Theorem 3 still exhibits a gap compared with UCB with known noise distribution when Delta is large. Do you think this gap is essential (unavoidable for any algorithms), or can be closed using some other approaches?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}