Recurrent Submodular Welfare and Matroid Blocking Semi-Bandits

1 Introduction . Despite the large number of variants of the stochastic multi-armed bandits ( MAB ) model [ 46 , 33 ] that have been introduced [ 8 , 34 ] , the majority of the results comply with the common assumption that playing an action does not alter the environment , namely , the reward distributions of the subsequent rounds ( with notable exceptions discussed below ) . Only recently , researchers have focused their attention on settings where temporal dependencies of specific structure are imposed between the player ’ s actions and the reward distributions [ 27 , 10 , 7 , 39 , 6 ] . In [ 27 ] , Kleinberg and Immorlica consider the setting of recharging bandits , where the expected reward of each arm is a concave and weakly increasing function of the time passed since its last play , modeling in that way scenarios of local performance loss . In a similar spirit , Basu et al . [ 7 ] consider the problem of blocking bandits , in which case once an arm is played at some round , it can not be played again ( i.e. , it becomes blocked ) for a fixed number of consecutive rounds . Notice that all the aforementioned examples are variations of the stochastic MAB setting , where the decision maker plays ( at most ) one arm per time step . When combinatorial constraints and time dynamics come together , the result is a much richer and more challenging setting , precisely because their interplay creates a complex dynamical structure . Indeed , in the standard combinatorial bandits setting [ 11 ] , the optimal solution in hindsight is to consistently play the feasible subset of arms of maximum expected reward . However , in the presence 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of local temporal constraints on the arms , an optimal ( or even suboptimal ) solution can not be trivially characterized– a fact that significantly complicates the analysis , both from the algorithmic as well as from the learning perspective . In this work , we study the following bandit setting– a common generalization of matroid bandits , introduced by Kveton et al . [ 30 ] , and blocking bandits [ 7 ] : Problem 1.1 ( Matroid Blocking Semi-Bandits ( MBB ) ) . We consider a set A of k arms , a matroid M = ( A , I ) , and an unknown time horizon of T rounds . Each arm i 2 A is associated with an unknown bounded reward distribution of mean µi , and with a known deterministic delay di , such that whenever an action i is played at some round , it can not be played again for the next di 1 rounds . At each time step , the player pulls a subset of the available ( i.e. , not blocked ) arms restricted to be an independent set of M. Subsequently , she observes the reward realization of each arm played ( semi-bandit feedback ) and collects their sum as the reward for this round . The goal of the player is to maximize her expected cumulative reward over T rounds . The above model captures a number of applications , varying from team formation to ad placement , when arms represent actions that can not be played repeatedly without restriction . As a concrete example , consider a recommendation system that repeatedly suggests a variety of products ( e.g. , songs , movies , books ) to a user . The need for diversity on the collection of suggested products ( arms ) , to capture different aspects of user ’ s preferences , can be modeled as a linear matroid . Further , the blocking constraints preclude the incessant recommendation of the same product ( which can be detrimental , as the product might be perceived as a “ spam ” ) , while the maximum rate of recommendation ( controlled by the delay ) might depend on factors such as popularity , promotion and more . Finally , the expected reward of each product is the probability of purchasing ( or clicking ) . From a technical viewpoint , the MBB problem is already NP-hard for the simple case of a uniform rank-1 matroid ( see Theorem 2.1 in [ 43 ] ) , even in the full-information setting , where the reward distributions are known to the player a priori . The natural common generalization of the algorithms in [ 7 , 30 ] , computes and plays , at each time step , an independent set of maximum mean reward consisting of the available elements . While the above strategy is a ( 1 1/e ) -approximation asymptotically ( that is , for T ! 1 ) for partition matroids , unfortunately , it only guarantees a 1/2-approximation for general matroids [ 1 ] and this guarantee is tight ( see Appendix E for an example ) . A natural question that arises is whether a ( 1 1/e ) -approximation is possible for any matroid . The main result of this paper shows that this is indeed possible . Along the way , we identify that the key insight ( and also the weak point of the naive 1/2-approximation ) is the underlying diminishing returns property hidden in the matroid structure . In particular , we discover an interesting connection of MBB to the following problem of interest in its own right : Problem 1.2 ( Recurrent Submodular Welfare ( RSW ) ) . We consider a monotone ( non-decreasing ) submodular function f : 2A ! R 0 over a universe A and a time horizon T . At each round t 2 [ T ] we choose a subset At ✓ A and collect a reward f ( At ) . However , using an element i 2 A at some round t 2 [ T ] makes it unavailable ( i.e. , blocked ) for a fixed and known number of di 1 subsequent rounds , namely , during the interval [ t , t+di 1 ] . The objective is to maximize P t2 [ T ] f ( At ) , subject to the blocking constraints , within a ( potentially unknown ) time horizon T . For the above model , which can be thought of as a variant of Submodular Welfare Maximization [ 47 ] , we provide an efficient randomized ( 1 1/e ) -approximation ( asymptotically ) , accompanied by a matching hardness result . Note that the RSW problem is a very natural model , capturing applications of submodular maximization in repeating scenarios , where the elements can not be constantly used without restriction . As an example , consider the process of renting goods to a stream of customers with identical submodular utility functions modeling their satisfaction . As we show , our approach for the RSW problem immediately implies an algorithm of the same approximation guarantee for the full-information case of MBB and , additionally , it has important implications for the bandit setting , where the reward distributions are initially unknown . The standard goal in this case is to provide a ( sublinear in the time horizon ) upper bound on the regret , namely , the difference between the expected reward of a bandit algorithm and a ( near- ) optimal algorithm , due to the initial lack of knowledge of the former1 . 1In fact , we upper bound the ( 1 1/e ) - ( approximate ) regret , defined as the difference between ( 1 1/e ) OPT ( T ) and the expected reward collected by a bandit algorithm . The notion of ↵-regret is widely used in the combinatorial bandits literature [ 15 , 48 ] for combinatorial problems where an efficient algorithm 1.1 Related Work . A recent line of research focuses on non-stationary models in the case where each reward distribution is a special function of the player ’ s actions [ 10 , 39 , 6 ] . In [ 7 ] , Basu et al . provide a greedy ( 1 1/e ) approximation for the full-information case of the blocking bandits problem ( a special case of the MBB model for a uniform rank-1 matroid ) . As we have already mentioned , generalizing their strategy to the MBB problem fails to provide the same guarantee for general matroids . In the bandit setting , where the reward distributions are initially unknown , the authors have to overcome the burden of characterizing a ( sub ) optimal solution , where the rate of mean collected reward exhibits significant fluctuations over time . The key insight is to observe that every time the full-information algorithm plays an arm , its bandit variant , which relies on estimations of the mean rewards , has at least one chance of playing the same arm . However , this key coupling argument , that enables sublinear regret bounds , becomes significantly more involved in the presence of matroid constraints . In [ 27 ] , Kleinberg and Immorlica study the case of recharging bandits . Their approach first computes the “ optimal ” playing frequency 1/xi of each arm i via a mathematical formulation . In order to play each arm with this frequency , they develop the technique of interleaved rounding , where they associate each arm i with a sequence of real numbers { ( ↵i+k ) /xi } k2N , with ↵i ⇠ U [ 0 , 1 ] . Then , the arms are played sequentially in the same order they appear on the real line . This novel rounding technique exhibits reduced variance and , thus , an improved approximation guarantee comparing to other natural approaches such as independent randomized rounding . The MBB model is also related to the literature on periodic scheduling [ 5 , 4 ] . In [ 43 ] , Sgall et al . consider the problem of periodically scheduling jobs on a set of machines . Each job is associated with a processing time , during which it occupies the machine it is executed on , a vacation time , namely , a minimum time required after its completion in order to be rescheduled , and a reward . It is not hard to see that the case of unit processing times is a special case of MBB with a uniform matroid of rank equal to the number of machines , under the objective of maximizing the total reward . Further , it is known [ 7 ] that the rank-1 case of MBB generalizes the Pinwheel Scheduling problem [ 24 ] : Given k colors associated a set of integers { di } i2 [ k ] , such that i2 [ k ] 1/di = 1 , decide whether there is a coloring of the natural numbers ⌫ : N ! [ k ] such that every color i 2 [ k ] appears at least once every di numbers . As it is proved in [ 25 ] , the above problem does not admit a pseudopolynomial time algorithm unless SAT can be solved by a randomized algorithm in expected quasi-polynomial time . In a concurrent work [ 1 ] , the authors consider the blocking bandit model in a generic combinatorial setting and under stochastic delays . As they show , the greedy algorithm that plays at each time the maximum feasible subset of available arms is a O ( 1 ) -approximation for downward-closed set systems . However , when specialized to matroids , they can not do better than a 1/2-approximation . We need new ideas to reach a ( 1 1/e ) -approximation algorithm and associated regret guarantees for the rich class of matroid bandits . Our work is related to the line of research regarding bandit optimization of submodular functions ( see [ 13 , 20 ] and references therein ) . We refer the reader to Appendix A for additional related work on non-stationary bandits , combinatorial bandits , and submodular welfare maximization .

Summary: This paper considers the problem of matroid semi-bandits with an additional condition that each arm becomes unavailable for a fixed duration after it is played. The paper first considers an "offline" version of the problem where the reward for each arm is already known and the goal is to design an optimal schedule for playing the arms. A randomized greedy algorithm is proposed which is (1-1/e)-competitive to an optimal algorithm. The paper then shows that this algorithm can be extended to the `online' version of the problem when the rewards are unknown. This is because the set of arms played in this greedy schedule only depends on the delays and does not depend on the rewards. This online algorithm suffers an additional regret of O(K\sqrt{T} + K^2) in comparison to the best offline solution.