Recurrent Submodular Welfare and Matroid Blocking Semi-Bandits

1 Introduction . Despite the large number of variants of the stochastic multi-armed bandits ( MAB ) model [ 46 , 33 ] that have been introduced [ 8 , 34 ] , the majority of the results comply with the common assumption that playing an action does not alter the environment , namely , the reward distributions of the subsequent rounds ( with notable exceptions discussed below ) . Only recently , researchers have focused their attention on settings where temporal dependencies of specific structure are imposed between the player ’ s actions and the reward distributions [ 27 , 10 , 7 , 39 , 6 ] . In [ 27 ] , Kleinberg and Immorlica consider the setting of recharging bandits , where the expected reward of each arm is a concave and weakly increasing function of the time passed since its last play , modeling in that way scenarios of local performance loss . In a similar spirit , Basu et al . [ 7 ] consider the problem of blocking bandits , in which case once an arm is played at some round , it can not be played again ( i.e. , it becomes blocked ) for a fixed number of consecutive rounds . Notice that all the aforementioned examples are variations of the stochastic MAB setting , where the decision maker plays ( at most ) one arm per time step . When combinatorial constraints and time dynamics come together , the result is a much richer and more challenging setting , precisely because their interplay creates a complex dynamical structure . Indeed , in the standard combinatorial bandits setting [ 11 ] , the optimal solution in hindsight is to consistently play the feasible subset of arms of maximum expected reward . However , in the presence 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of local temporal constraints on the arms , an optimal ( or even suboptimal ) solution can not be trivially characterized– a fact that significantly complicates the analysis , both from the algorithmic as well as from the learning perspective . In this work , we study the following bandit setting– a common generalization of matroid bandits , introduced by Kveton et al . [ 30 ] , and blocking bandits [ 7 ] : Problem 1.1 ( Matroid Blocking Semi-Bandits ( MBB ) ) . We consider a set A of k arms , a matroid M = ( A , I ) , and an unknown time horizon of T rounds . Each arm i 2 A is associated with an unknown bounded reward distribution of mean µi , and with a known deterministic delay di , such that whenever an action i is played at some round , it can not be played again for the next di 1 rounds . At each time step , the player pulls a subset of the available ( i.e. , not blocked ) arms restricted to be an independent set of M. Subsequently , she observes the reward realization of each arm played ( semi-bandit feedback ) and collects their sum as the reward for this round . The goal of the player is to maximize her expected cumulative reward over T rounds . The above model captures a number of applications , varying from team formation to ad placement , when arms represent actions that can not be played repeatedly without restriction . As a concrete example , consider a recommendation system that repeatedly suggests a variety of products ( e.g. , songs , movies , books ) to a user . The need for diversity on the collection of suggested products ( arms ) , to capture different aspects of user ’ s preferences , can be modeled as a linear matroid . Further , the blocking constraints preclude the incessant recommendation of the same product ( which can be detrimental , as the product might be perceived as a “ spam ” ) , while the maximum rate of recommendation ( controlled by the delay ) might depend on factors such as popularity , promotion and more . Finally , the expected reward of each product is the probability of purchasing ( or clicking ) . From a technical viewpoint , the MBB problem is already NP-hard for the simple case of a uniform rank-1 matroid ( see Theorem 2.1 in [ 43 ] ) , even in the full-information setting , where the reward distributions are known to the player a priori . The natural common generalization of the algorithms in [ 7 , 30 ] , computes and plays , at each time step , an independent set of maximum mean reward consisting of the available elements . While the above strategy is a ( 1 1/e ) -approximation asymptotically ( that is , for T ! 1 ) for partition matroids , unfortunately , it only guarantees a 1/2-approximation for general matroids [ 1 ] and this guarantee is tight ( see Appendix E for an example ) . A natural question that arises is whether a ( 1 1/e ) -approximation is possible for any matroid . The main result of this paper shows that this is indeed possible . Along the way , we identify that the key insight ( and also the weak point of the naive 1/2-approximation ) is the underlying diminishing returns property hidden in the matroid structure . In particular , we discover an interesting connection of MBB to the following problem of interest in its own right : Problem 1.2 ( Recurrent Submodular Welfare ( RSW ) ) . We consider a monotone ( non-decreasing ) submodular function f : 2A ! R 0 over a universe A and a time horizon T . At each round t 2 [ T ] we choose a subset At ✓ A and collect a reward f ( At ) . However , using an element i 2 A at some round t 2 [ T ] makes it unavailable ( i.e. , blocked ) for a fixed and known number of di 1 subsequent rounds , namely , during the interval [ t , t+di 1 ] . The objective is to maximize P t2 [ T ] f ( At ) , subject to the blocking constraints , within a ( potentially unknown ) time horizon T . For the above model , which can be thought of as a variant of Submodular Welfare Maximization [ 47 ] , we provide an efficient randomized ( 1 1/e ) -approximation ( asymptotically ) , accompanied by a matching hardness result . Note that the RSW problem is a very natural model , capturing applications of submodular maximization in repeating scenarios , where the elements can not be constantly used without restriction . As an example , consider the process of renting goods to a stream of customers with identical submodular utility functions modeling their satisfaction . As we show , our approach for the RSW problem immediately implies an algorithm of the same approximation guarantee for the full-information case of MBB and , additionally , it has important implications for the bandit setting , where the reward distributions are initially unknown . The standard goal in this case is to provide a ( sublinear in the time horizon ) upper bound on the regret , namely , the difference between the expected reward of a bandit algorithm and a ( near- ) optimal algorithm , due to the initial lack of knowledge of the former1 . 1In fact , we upper bound the ( 1 1/e ) - ( approximate ) regret , defined as the difference between ( 1 1/e ) OPT ( T ) and the expected reward collected by a bandit algorithm . The notion of ↵-regret is widely used in the combinatorial bandits literature [ 15 , 48 ] for combinatorial problems where an efficient algorithm 1.1 Related Work . A recent line of research focuses on non-stationary models in the case where each reward distribution is a special function of the player ’ s actions [ 10 , 39 , 6 ] . In [ 7 ] , Basu et al . provide a greedy ( 1 1/e ) approximation for the full-information case of the blocking bandits problem ( a special case of the MBB model for a uniform rank-1 matroid ) . As we have already mentioned , generalizing their strategy to the MBB problem fails to provide the same guarantee for general matroids . In the bandit setting , where the reward distributions are initially unknown , the authors have to overcome the burden of characterizing a ( sub ) optimal solution , where the rate of mean collected reward exhibits significant fluctuations over time . The key insight is to observe that every time the full-information algorithm plays an arm , its bandit variant , which relies on estimations of the mean rewards , has at least one chance of playing the same arm . However , this key coupling argument , that enables sublinear regret bounds , becomes significantly more involved in the presence of matroid constraints . In [ 27 ] , Kleinberg and Immorlica study the case of recharging bandits . Their approach first computes the “ optimal ” playing frequency 1/xi of each arm i via a mathematical formulation . In order to play each arm with this frequency , they develop the technique of interleaved rounding , where they associate each arm i with a sequence of real numbers { ( ↵i+k ) /xi } k2N , with ↵i ⇠ U [ 0 , 1 ] . Then , the arms are played sequentially in the same order they appear on the real line . This novel rounding technique exhibits reduced variance and , thus , an improved approximation guarantee comparing to other natural approaches such as independent randomized rounding . The MBB model is also related to the literature on periodic scheduling [ 5 , 4 ] . In [ 43 ] , Sgall et al . consider the problem of periodically scheduling jobs on a set of machines . Each job is associated with a processing time , during which it occupies the machine it is executed on , a vacation time , namely , a minimum time required after its completion in order to be rescheduled , and a reward . It is not hard to see that the case of unit processing times is a special case of MBB with a uniform matroid of rank equal to the number of machines , under the objective of maximizing the total reward . Further , it is known [ 7 ] that the rank-1 case of MBB generalizes the Pinwheel Scheduling problem [ 24 ] : Given k colors associated a set of integers { di } i2 [ k ] , such that i2 [ k ] 1/di = 1 , decide whether there is a coloring of the natural numbers ⌫ : N ! [ k ] such that every color i 2 [ k ] appears at least once every di numbers . As it is proved in [ 25 ] , the above problem does not admit a pseudopolynomial time algorithm unless SAT can be solved by a randomized algorithm in expected quasi-polynomial time . In a concurrent work [ 1 ] , the authors consider the blocking bandit model in a generic combinatorial setting and under stochastic delays . As they show , the greedy algorithm that plays at each time the maximum feasible subset of available arms is a O ( 1 ) -approximation for downward-closed set systems . However , when specialized to matroids , they can not do better than a 1/2-approximation . We need new ideas to reach a ( 1 1/e ) -approximation algorithm and associated regret guarantees for the rich class of matroid bandits . Our work is related to the line of research regarding bandit optimization of submodular functions ( see [ 13 , 20 ] and references therein ) . We refer the reader to Appendix A for additional related work on non-stationary bandits , combinatorial bandits , and submodular welfare maximization .

Summary: The authors consider the problem of Matroid Blocking Semi-Bandits. In this setting, there are $k$ arms, an unknown time horizon $T$ and a matroid dictating which sets of arms can be pulled at each step. Pulling each arm yields a reward, which comes from an unknown probability distribution with mean $\mu_i$. Moreover, each arm $i$ remains blocked after use for $d_i$ steps. Observing only the rewards from the used arms, the goal is to maximise the sum of all rewards collected over the time horizon $T$.   There are 3 main results, which can be better explained by a minor detour in the full information setting (where the $\mu_i$ are known). In this setting, the first approach one would follow is to ignore the $d_i$’s and just chose the independent set maximising the current rewards, among the arms that are not blocked. However, this is not optimal (for general matroids): the catch is that the underlying matroid indirectly affects the rewards of each arm, by restricting which other arms can be used together. To get around this, the authors establish a reduction between this setting and what they call the Recurrent Submodular Welfare (RSW), which entails maximising a monotone submodular function over $T$ steps, but where each element has a similar delay $d_i$ before it can be selected again.   The RSW problem can be solved by a clever randomisation: rather than choosing each element with probability $1/d_i$, the authors interleave the elements by adding a random delay $r_i \in [0,1]$ and selecting the item whenever the interval [t/d_i + r_i, (t+1)/d_i + r_i] contains an integer. This simple step takes care of most of the inefficiency and the unknown time horizon $T$ is also handled: essentially, just the last step might be suboptimal and this effect decreases with $T$. This algorithm is optimal for this setting, with a $1-1/e$ approximation.  The full information setting can be handled by selecting arms a subset of arms that are available in the RSW case, but doing it greedily and stopping once an independent set is found, to satisfy that hard matroid constraints. The reduction is completed by using the weighted rank function, which converts an additive function with matroid constraints into a monotone submodular one. As such, this algorithm is also optimal with an approximation of $1-1/e$ (and some minor terms relating to $\max d_i$).  The final step into the bandit setting can be performed because the learning of the $\mu_i$ and the random selection of elements are decoupled: the RSW interleaving technique is ‘static’ and does not update its schedule as the algorithm runs. Therefore, a modification of UCB can be used to balance learning the $\mu_i$’s with exploiting them, while observing the schedule used by RSW. This also yields an optimal (1-1/e) approximation (with an additional  $O(k\sqrt{T \log T} + k^2  + \max d_i \cdot r)$ additive loss, where $r$ is the matroid rank).