Recurrent Submodular Welfare and Matroid Blocking Semi-Bandits

1 Introduction . Despite the large number of variants of the stochastic multi-armed bandits ( MAB ) model [ 46 , 33 ] that have been introduced [ 8 , 34 ] , the majority of the results comply with the common assumption that playing an action does not alter the environment , namely , the reward distributions of the subsequent rounds ( with notable exceptions discussed below ) . Only recently , researchers have focused their attention on settings where temporal dependencies of specific structure are imposed between the player ’ s actions and the reward distributions [ 27 , 10 , 7 , 39 , 6 ] . In [ 27 ] , Kleinberg and Immorlica consider the setting of recharging bandits , where the expected reward of each arm is a concave and weakly increasing function of the time passed since its last play , modeling in that way scenarios of local performance loss . In a similar spirit , Basu et al . [ 7 ] consider the problem of blocking bandits , in which case once an arm is played at some round , it can not be played again ( i.e. , it becomes blocked ) for a fixed number of consecutive rounds . Notice that all the aforementioned examples are variations of the stochastic MAB setting , where the decision maker plays ( at most ) one arm per time step . When combinatorial constraints and time dynamics come together , the result is a much richer and more challenging setting , precisely because their interplay creates a complex dynamical structure . Indeed , in the standard combinatorial bandits setting [ 11 ] , the optimal solution in hindsight is to consistently play the feasible subset of arms of maximum expected reward . However , in the presence 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of local temporal constraints on the arms , an optimal ( or even suboptimal ) solution can not be trivially characterized– a fact that significantly complicates the analysis , both from the algorithmic as well as from the learning perspective . In this work , we study the following bandit setting– a common generalization of matroid bandits , introduced by Kveton et al . [ 30 ] , and blocking bandits [ 7 ] : Problem 1.1 ( Matroid Blocking Semi-Bandits ( MBB ) ) . We consider a set A of k arms , a matroid M = ( A , I ) , and an unknown time horizon of T rounds . Each arm i 2 A is associated with an unknown bounded reward distribution of mean µi , and with a known deterministic delay di , such that whenever an action i is played at some round , it can not be played again for the next di 1 rounds . At each time step , the player pulls a subset of the available ( i.e. , not blocked ) arms restricted to be an independent set of M. Subsequently , she observes the reward realization of each arm played ( semi-bandit feedback ) and collects their sum as the reward for this round . The goal of the player is to maximize her expected cumulative reward over T rounds . The above model captures a number of applications , varying from team formation to ad placement , when arms represent actions that can not be played repeatedly without restriction . As a concrete example , consider a recommendation system that repeatedly suggests a variety of products ( e.g. , songs , movies , books ) to a user . The need for diversity on the collection of suggested products ( arms ) , to capture different aspects of user ’ s preferences , can be modeled as a linear matroid . Further , the blocking constraints preclude the incessant recommendation of the same product ( which can be detrimental , as the product might be perceived as a “ spam ” ) , while the maximum rate of recommendation ( controlled by the delay ) might depend on factors such as popularity , promotion and more . Finally , the expected reward of each product is the probability of purchasing ( or clicking ) . From a technical viewpoint , the MBB problem is already NP-hard for the simple case of a uniform rank-1 matroid ( see Theorem 2.1 in [ 43 ] ) , even in the full-information setting , where the reward distributions are known to the player a priori . The natural common generalization of the algorithms in [ 7 , 30 ] , computes and plays , at each time step , an independent set of maximum mean reward consisting of the available elements . While the above strategy is a ( 1 1/e ) -approximation asymptotically ( that is , for T ! 1 ) for partition matroids , unfortunately , it only guarantees a 1/2-approximation for general matroids [ 1 ] and this guarantee is tight ( see Appendix E for an example ) . A natural question that arises is whether a ( 1 1/e ) -approximation is possible for any matroid . The main result of this paper shows that this is indeed possible . Along the way , we identify that the key insight ( and also the weak point of the naive 1/2-approximation ) is the underlying diminishing returns property hidden in the matroid structure . In particular , we discover an interesting connection of MBB to the following problem of interest in its own right : Problem 1.2 ( Recurrent Submodular Welfare ( RSW ) ) . We consider a monotone ( non-decreasing ) submodular function f : 2A ! R 0 over a universe A and a time horizon T . At each round t 2 [ T ] we choose a subset At ✓ A and collect a reward f ( At ) . However , using an element i 2 A at some round t 2 [ T ] makes it unavailable ( i.e. , blocked ) for a fixed and known number of di 1 subsequent rounds , namely , during the interval [ t , t+di 1 ] . The objective is to maximize P t2 [ T ] f ( At ) , subject to the blocking constraints , within a ( potentially unknown ) time horizon T . For the above model , which can be thought of as a variant of Submodular Welfare Maximization [ 47 ] , we provide an efficient randomized ( 1 1/e ) -approximation ( asymptotically ) , accompanied by a matching hardness result . Note that the RSW problem is a very natural model , capturing applications of submodular maximization in repeating scenarios , where the elements can not be constantly used without restriction . As an example , consider the process of renting goods to a stream of customers with identical submodular utility functions modeling their satisfaction . As we show , our approach for the RSW problem immediately implies an algorithm of the same approximation guarantee for the full-information case of MBB and , additionally , it has important implications for the bandit setting , where the reward distributions are initially unknown . The standard goal in this case is to provide a ( sublinear in the time horizon ) upper bound on the regret , namely , the difference between the expected reward of a bandit algorithm and a ( near- ) optimal algorithm , due to the initial lack of knowledge of the former1 . 1In fact , we upper bound the ( 1 1/e ) - ( approximate ) regret , defined as the difference between ( 1 1/e ) OPT ( T ) and the expected reward collected by a bandit algorithm . The notion of ↵-regret is widely used in the combinatorial bandits literature [ 15 , 48 ] for combinatorial problems where an efficient algorithm 1.1 Related Work . A recent line of research focuses on non-stationary models in the case where each reward distribution is a special function of the player ’ s actions [ 10 , 39 , 6 ] . In [ 7 ] , Basu et al . provide a greedy ( 1 1/e ) approximation for the full-information case of the blocking bandits problem ( a special case of the MBB model for a uniform rank-1 matroid ) . As we have already mentioned , generalizing their strategy to the MBB problem fails to provide the same guarantee for general matroids . In the bandit setting , where the reward distributions are initially unknown , the authors have to overcome the burden of characterizing a ( sub ) optimal solution , where the rate of mean collected reward exhibits significant fluctuations over time . The key insight is to observe that every time the full-information algorithm plays an arm , its bandit variant , which relies on estimations of the mean rewards , has at least one chance of playing the same arm . However , this key coupling argument , that enables sublinear regret bounds , becomes significantly more involved in the presence of matroid constraints . In [ 27 ] , Kleinberg and Immorlica study the case of recharging bandits . Their approach first computes the “ optimal ” playing frequency 1/xi of each arm i via a mathematical formulation . In order to play each arm with this frequency , they develop the technique of interleaved rounding , where they associate each arm i with a sequence of real numbers { ( ↵i+k ) /xi } k2N , with ↵i ⇠ U [ 0 , 1 ] . Then , the arms are played sequentially in the same order they appear on the real line . This novel rounding technique exhibits reduced variance and , thus , an improved approximation guarantee comparing to other natural approaches such as independent randomized rounding . The MBB model is also related to the literature on periodic scheduling [ 5 , 4 ] . In [ 43 ] , Sgall et al . consider the problem of periodically scheduling jobs on a set of machines . Each job is associated with a processing time , during which it occupies the machine it is executed on , a vacation time , namely , a minimum time required after its completion in order to be rescheduled , and a reward . It is not hard to see that the case of unit processing times is a special case of MBB with a uniform matroid of rank equal to the number of machines , under the objective of maximizing the total reward . Further , it is known [ 7 ] that the rank-1 case of MBB generalizes the Pinwheel Scheduling problem [ 24 ] : Given k colors associated a set of integers { di } i2 [ k ] , such that i2 [ k ] 1/di = 1 , decide whether there is a coloring of the natural numbers ⌫ : N ! [ k ] such that every color i 2 [ k ] appears at least once every di numbers . As it is proved in [ 25 ] , the above problem does not admit a pseudopolynomial time algorithm unless SAT can be solved by a randomized algorithm in expected quasi-polynomial time . In a concurrent work [ 1 ] , the authors consider the blocking bandit model in a generic combinatorial setting and under stochastic delays . As they show , the greedy algorithm that plays at each time the maximum feasible subset of available arms is a O ( 1 ) -approximation for downward-closed set systems . However , when specialized to matroids , they can not do better than a 1/2-approximation . We need new ideas to reach a ( 1 1/e ) -approximation algorithm and associated regret guarantees for the rich class of matroid bandits . Our work is related to the line of research regarding bandit optimization of submodular functions ( see [ 13 , 20 ] and references therein ) . We refer the reader to Appendix A for additional related work on non-stationary bandits , combinatorial bandits , and submodular welfare maximization .

Summary: The paper studies an extension of the matroid bandit problem where each arm $i$ is blocked for a known $d_i$ rounds after it is played, and this is referred to as the matroid blocking semi-bandit (MBB) problem. When the arm distributions are known, the authors reduce the optimization problem to a recurrent submodular welfare (RSW) problem. The authors first propose a 1-1/e approximation to the RSW problem, then show that for MBB with unknown reward distributions, using the upper confidence bounds (UCB) instead of the known mean values would give a sublinear approximation regret with approximation ratio of 1-1/e. The main contribution of the paper is to achieve 1-1/e approximation, while an easy adaptation from some other work only gives 1/2 approximation.