Multi-Armed Bandits with Bounded Arm-Memory: Near-Optimal Guarantees for Best-Arm Identification and Regret Minimization

1 δ ) ) optimal sample complexity . 1 Introduction . The Stochastic Multi-armed Bandit ( MAB ) problem is a classical framework used to capture decisionmaking in uncertain environments . In this model , a decision-maker is faced with n choices ( called arms ) and has to sequentially choose one of the n arms ( referred to as pulling an arm ) . Based on the pulled arm , the decision-maker gets a reward drawn from a corresponding reward distribution unknown to the decision-maker . Starting with the seminal work of Robbins [ 26 ] , MAB has been extensively studied with one of the following two goals : regret minimization and best-arm identification . See , e.g . [ 7 ] for a textbook treatment of the area . In addition to being theoretically interesting , MAB also finds numerous practical applications in diverse areas , including online advertising [ 31 ] , crowd-sourcing [ 32 ] , and clinical trials [ 9 ] . Hence , the study of MAB and its variants is of paramount interest in multiple fields , including online learning and reinforcement learning . In many applications of the MAB problem , the number of arms ( set of advertisements , crowd-workers , etc . ) could be huge . Thus , the MAB problem with a large set of arms has received significant attention in recent years . Starting with the formative work of Berry et al . [ 6 ] and Herschkorn et al . [ 16 ] which studied the MAB problem with infinitely many arms , to the more recent work of Kleinberg et al . [ 20 ] which studies the MAB problem where the set of arms form a metric space , this variant has remained an active and prominent area of research . Large number of arms may enforce a constraint that the algorithm may not be able to store all the arms in the memory simultaneously . Additionally , as is common in these applications , the arms could arrive online , i.e. , the algorithm may not have access to the entire set of arms at the beginning . The streaming model , first formalized in the pioneering work of Alon et al . [ 3 ] , has been developed to handle data streams where a huge amount of data arrives online and an algorithm has access ∗A part of this work was done when the author was an undergraduate summer intern at IISc , Bangalore . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to only a limited amount of memory . We refer the readers to [ 23 , 2 ] for a detailed survey of streaming algorithms . With the ever-increasing amount of data available to any machine learning ( ML ) algorithm , several salient areas of ML are being studied under streaming constraints ; e.g. , active learning [ 35 , 29 ] , lifelong learning [ 33 ] , and identifying correlations in multivariate data [ 11 ] ( see the recent survey by Gomes et al . [ 15 ] for further discussions ) . In this work , we study the classical stochastic MAB problem under streaming constraints . This model has recently been studied in [ 27 , 21 , 4 ] . In particular , we study a setting where we are given a set of n arms arriving one-by-one , and an algorithm can store only a fixed number of arms m < n in the memory . The set of arms stored in the memory is known as arm-memory and m denotes the upper bound on the number of arms that can be stored in the memory at any time step . In this model , at any time step an arm not in the current arm-memory can not be pulled . Further , for any arm currently not in the arm-memory , we assume we do not know anything about the reward statistics corresponding to that arm until that arm is read into arm-memory and subsequently pulled . Note that the classical MAB algorithms such as UCB1 [ 5 ] and Thompson Sampling [ 30 ] for regret minimization , and Median-Elimination [ 13 ] for best-arm identification are not suited for this streaming setting as they require all arms to be present in the arm-memory at the start of the algorithm . Our goal is to study the interplay between arm-memory size vs. expected regret and arm-memory size vs. sample complexity , respectively . An algorithm for our setting works as follows : at any time step t , the algorithm can only pull an arm that is currently in arm-memory and update its reward statistics . Then , the algorithm may choose to discard some of the arms that are currently in the arm-memory and read new arms into the arm-memory before proceeding to the next time-step to pull an arm from the arm-memory . In streaming terminology , algorithms that can not read back an arm into the arm-memory that was previously discarded from the arm-memory , are called single-pass or one-pass algorithms . On the other hand , algorithms that are allowed to read back an arm into the arm-memory that was previously discarded from the arm-memory , are called multi-pass algorithms . We note that our algorithms are designed for the more challenging single-pass setting , which has also been the focus of other prominent areas in machine learning , for example , Rai et al . [ 25 ] focus on designing one-pass SVMs when the data arrives in a stream , whereas Carvalho and Cohen [ 8 ] design single-pass algorithms for online learning . 1.1 Our Contribution . We study MAB under bounded arm-memory , where n arms arrive in a stream and at most m < n arms can be stored in the arm-memory at any time . We study the trade-off between arm-memory size vs. expected regret , and arm-memory size vs. sample complexity . Regret minimization : Our first result , in Section 3 , settles an open question stated in both [ 21 ] and [ 27 ] pertaining to the lower bound on the expected cumulative regret in this model . Using information-theoretic machinery related to KL-divergence , we show that any single-pass algorithm in this model will incur an expected regret of Ω ( n1/3 · T 2/3/m7/3 ) . Interestingly , this result holds for any m < n , which shows that even if the algorithm is allowed to store n− 1 arms in the arm-memory at any time , we can not hope to obtain a better regret guarantee in terms of T . This lower bound almost matches with the Õ ( n1/3T 2/3 ) bound on the expected cumulative regret , obtained by the standard uniform-exploration algorithm where we store m = 2 arms . We note here that the uniformexploration algorithm with m = 2 can be trivially extended to an algorithm that stores m > 2 arms in the memory and achieves a regret guarantee that matches the lower bound by storing the 2 arms required by the uniform-exploration algorithm and filling the remaining m− 2 sized arm-memory with arbitrary arms that would have otherwise been discarded . Since this is already optimal for the single-pass setting , we can not hope to get a better regret guarantee in terms of T , even if we store n− 1 arms in the memory . This shows an interesting dichotomy between m = n− 1 and m = n , as one can obtain expected cumulative regret of Õ ( √ nT ) by standard UCB1 algorithm [ 5 ] , where we are allowed to store n arms . Our lower bound holds even when the arms arrive in a uniformly random sequence . Best-arm identification : In Section 4 , we propose an ( ε , δ ) -PAC streaming algorithm that takes r as input and stores r + 1 arms in the arm-memory at any time and has sample complexity O ( nε2 · ( ilog ( r ) ( n ) + ln ( 1δ ) ) ) 2 . In particular , when r = log∗ n , our algorithm achieves the optimal worstcase sample complexity O ( n ε2 log ( 1 δ ) ) for any best-arm identification algorithm [ 13 ] . This problem was also studied in recent STOC ’ 20 paper by Assadi and Wang [ 4 ] where they proposed an ( ε , δ ) -PAC algorithm with optimal sample complexity and O ( log∗ n ) arm-memory size . However , we show that due to an error in their analysis , their claim is incorrect and their algorithm is not ( ε , δ ) -PAC . In Appendix E , we construct a family of input instances for which their algorithm will output a non-ε-best arm with probability significantly larger than δ . In Section 4 , we show that the special case of our algorithm ( when r = log∗ n ) indeed provides the guarantee claimed in [ 4 ] . 1.2 Related Work . In one of the earliest works on MAB with bounded arm-memory , Herschkorn et al . [ 16 ] study the infinite-armed MAB problem where only a single arm is stored in the arm-memory at any time step and an arm that is once discarded from the arm-memory can not be recalled . On the other hand , Cover [ 10 ] studied a different variant of bounded memory where the algorithm can remember the rewards of only a finite number of samples for each arm in a two-armed bandit model . The work of Lu and Lu [ 22 ] also studies this variant of bounded memory in the learning from experts setting , where the constraint is on the number of time steps for which the rewards can be stored by the algorithm and not on the number of arms . More recently , Kalvit and Zeevi [ 18 ] studied the infinite-armed bandit problem where each arm is associated with a type , and the number of types is finite . An algorithm knows the set of types but not the exact type of each arm . For this problem , they propose an algorithm that samples a finite number of arms and has O ( log T ) expected cumulative regret . Further , de Heide et al . [ 12 ] also study the infinite-armed bandit problem where there could be multiple optimal arms . They propose an algorithm and prove that it achieves O ( log T ) expected regret by sampling O ( log T ) number of arms . In our work , we focus on the finite-armed MAB problem with bounded arm-memory . Recently , the MAB problem where arms arrive in a stream and arm-memory is bounded , has been studied in both regret minimization and best-arm identification frameworks . For the regret minimization problem in the stochastic MAB setting , Peköz [ 24 ] showed that when m = 1 , a single-pass algorithm can have linear regret unless the reward distributions of arms satisfy some additional conditions . Liau et al . [ 21 ] and RoyChaudhuri and Kalyanakrishnan [ 27 ] study the stochastic MAB problem with bounded arm-memory with the goal of minimizing the expected cumulative regret over T time steps . The algorithms in both these works are multi-pass algorithms whereas our work focusses on single-pass algorithms . Liau et al . [ 21 ] propose an upper-confidence bound based algorithm with O ( 1 ) arm-memory size , which achieves an expected cumulative regret bound of O ( ∑ i 6=i∗ log ( ∆i ∆ ) log T ∆i ) , which is within log ( ∆i∆ ) factor of the UCB1 regret bound [ 5 ] . Here ∆ is the gap parameter , i.e. , the difference in the expected rewards of the best and the second-best arms , and ∆i is the difference in the expected rewards of the best arm and the i ’ th arm . However , they do not provide a worst-case regret bound for their algorithm , and hence , depending on the instance the regret of this algorithm may be very high . More precisely , if say ∆i is 1/T 2/3 , then the proven regret bound implies O ( T 2/3 ) expected cumulative regret . RoyChaudhuri and Kalyanakrishnan [ 27 ] propose an algorithmic framework , which is given m as input and uses a MAB algorithm as a black-box . When the MAB algorithm used is UCB1 , their algorithm achieves an expected regret of O ( nm+ n3/2m √ T log ( T/nm ) ) using Ω ( log T ) passes . The recent work of Xu and Zhao [ 34 ] also studies the regret minimization problem with bounded arm-memory constraints . However , they consider the setting with adversarial rewards and is different from the stochastic MAB setting studied in our paper . Recently , Assadi and Wang [ 4 ] studied the best-arm identification variant of this problem . First , they propose an algorithm for the best-coin identification problem ( equivalent to MAB with Bernoulli reward distributions ) , which keeps exactly one extra coin in the memory at any time and has optimal sample complexity . They further extend their algorithm to the top-k coins identification problem , which stores k coins in the memory and has optimal sample complexity . Crucially though , both algorithms assume that the gap parameter ∆ is known . Throughout our work , we deal with the case when ∆ is not known to the algorithm . Falahatgar et al . [ 14 ] also study the best-arm identification 2Here ilog ( r ) is iterated logarithm of order r ( see Section 1.3 ) . problem with bounded arm-memory and provide an algorithm with O ( 1 ) arm-memory size . However , they only provide a high probability guarantee on the sample complexity while additionally assuming that the arms arrive in a uniformly random sequence . Note that unlike [ 14 ] , the sample complexity of our algorithm is independent of the rewards observed during the sampling , and we do not require the assumption that the arms need to arrive in a uniformly random sequence . In a recent independent work , Jin et al . [ 17 ] proposed an ε-top-k arms identification algorithm with optimal sample complexity and arm-memory size of one . They also propose an algorithm that identifies the best-arm with high probability using arm-memory size equal to one , having optimal instance-dependent sample complexity , and using instance-dependent constant number of passes . We would like to note that their work provides high probability bounds on the sample complexity , whereas our work provides worst-case bounds on the same quantity .

Summary: This paper studies a variant of stochastic multi-armed bandits in which the learner doesn't have access to all the arms. The learner has a memory of size $m$, which means it can store and access only $m$ arms. The learner can discard some arms to free up space for new arms, which come in a stream. When an arm is discarded, it can never be accessed again.   The standard problems of regret minimization and best-arm identification are studied here; a regret lower bound for the former and an algorithm for the latter are given.  Let $T, n,$ and $m$ denote the time horizon, the number of arms, and the memory size, respectively. When $m=2$, a regret upper bound of $\widetilde{O}(n^{1/3}T^{2/3})$ was shown in previous work. This paper gives a matching lower bound up to logarithmic factors, and gives a lower bound of ${\Omega}(n^{1/3}T^{2/3}/m^{7/3})$ for all $m=1,2,\dots,n-1$. This is in sharp contrast with the $m=n$ case, which achieves $O(\sqrt{nT})$ regret.  For best-arm identification, the paper provides an algorithm with memory size $1 + \log^* (n)$ that pulls $O(n\log(1/\delta)/\varepsilon^2)$ arms and outputs an $\varepsilon$-optimal arm with probability $1-\delta$. This sample complexity is optimal even with memory size $n$. A similar result was shown in a previous work, but this paper claims that that proof had a bug.