Title: Bandit Task Assignment with Unknown Processing Time
Venue: NeurIPS 2023
Accepted: yes
Keywords: bandit

combinatorial semi-bandits

bandits with badget
Abstract: This study considers a novel problem setting, referred to as \textit{bandit task assignment}, that incorporates the processing time of each task in the bandit setting. In this problem setting, a player sequentially chooses a set of tasks to start so that the set of processing tasks satisfies a given combinatorial constraint. The reward and processing time for each task follow unknown distributions, values of which are revealed only after the task has been completed. The problem generalizes the stochastic combinatorial semi-bandit problem and the budget-constrained bandit problem. For this problem setting, we propose an algorithm based on upper confidence bounds~(UCB) combined with a phased-update approach. The proposed algorithm admits a gap-dependent regret upper bound of $O(MN(1/\Delta){\log T})$ and a gap-free regret upper bound of $\tilde{O}( \sqrt{MNT} )$, where $N$ is the number of the tasks, $M$ is the maximum number of tasks run at the same time, $T$ is the time horizon, and $\Delta$ is the gap between expected per-round rewards of the optimal and best suboptimal sets of tasks. These regret bounds nearly match lower bounds.

Bandit Task Assignment with Unknown Processing Time

Abstract

This study considers a novel problem setting, referred to as bandit task assignment , that incorporates the processing time of each task in the bandit setting. In this problem setting, a player sequentially chooses a set of tasks to start so that the set of processing tasks satisfies a given combinatorial constraint. The reward and processing time for each task follow unknown distributions, values of which are revealed only after the task has been completed. The problem generalizes the stochastic combinatorial semi-bandit problem and the budget-constrained bandit problem. For this problem setting, we propose an algorithm based on upper confidence bounds (UCB) combined with a phased-update approach. The proposed algorithm admits a gap-dependent regret upper bound of O ( MN (1 / ∆)log T ) and a gap-free regret upper bound of ˜ O ( √ MNT ) , where N is the number of the tasks, M is the maximum number of tasks run at the same time, T is the time horizon, and ∆ is the gap between expected per-round rewards of the optimal and best suboptimal sets of tasks. These regret bounds nearly match lower bounds.

1 Introduction

This paper introduces a new model of sequential decision-making that we refer to as the bandit task assignment problem. The goal of this model is to determine which tasks to perform sequentially so that the total reward will be maximized, while estimating the distribution of rewards and processing time for each task. For each round t = 1 , 2 , . . . , T , we choose a set A t of tasks to start. Each task i in A t will be completed in the ( t + c ti ) -th round, and we then obtain the reward r ti . The processing time c ti and the reward r ti follow unknown distributions independently for all t , and they are observed only when the task has been completed, i.e., only bandit feedback is available. In all rounds, the set of processing tasks is required to satisfy a given combinatorial constraint. That is, in each round, the player can start a set of new tasks that, together with the tasks still in progress, satisfies the given

constraint. The performance of the algorithm is evaluated by means of regret R T , which is defined as the difference between the cumulative rewards obtained with an optimal policy and with the algorithm. Our model allows an arbitrary constraint for a set of tasks that is expressed by A ⊆ { 0 , 1 } N , where N is the number of tasks. Each vector a = ( a 1 , a 2 , . . . , a N ) glyph[latticetop] ∈ A is interpreted as a feasible set of tasks A = { i ∈ [ N ] | a i = 1 } , where [ N ] := { 1 , 2 , . . . , N } .

Figure 1 illustrates an example of a feasible task assignment for the case of A = { a ∈ { 0 , 1 } 3 | ‖ a ‖ 1 ≤ 2 } , in which we choose which tasks to start from three candidates so that the number of processing tasks does not exceed two. Each arrow in this figure represents the interval of time to process a task. In this example, at round 2 , two tasks (Tasks 1 and 2) are still processing, and hence, we cannot start any new task. At the beginning of round 3 , we observe that Task 1 is completed, and it is then possible to start one new task. We note that the processing time c ti and the reword r ti of task i stated at time t are revealed only at the completion time ( t + c ti ) .

The model in this paper is a common generalization of stochastic combinatorial semi-bandit problems [25, 24, 34] and budget-constrained bandit problems [20, 15, 12, 37]. In fact, if the processing time c ti is always equal to 1 for any task i , the player can choose any set of A in every round and can observe the feedback immediately afterward, which coincides with the stochastic combinatorial semi-bandit problem. On the other hand, if we suppose that A = { a ∈ { 0 , 1 } N | ‖ a ‖ 1 ≤ 1 } , i.e., if only a single task can be performed in every round, the problem corresponds to stochastic N -armed bandits with a budget constraint [20, 15]. Indeed, by regarding c ti as the cost for choosing arm i , we can interpret the model as a problem with the budget constraint that the cumulative cost is at most T .

We note that our problem setting is similar to blocking bandits [8], in which each chosen arm is blocked , i.e., is unavailable for a certain period of time after it is chosen. However, they are different in the constraints on the actions in each round. For example, consider A = { a ∈ { 0 , 1 } N | ‖ a ‖ 1 ≤ K } . Then, in the bandit task assignment problem, we cannot start a new task when K tasks are already running. On the other hand, in the blocking bandit problem, the player can choose any K arms that are not blocked. Blocking bandits have been extended to contextual bandit models [9], semi-bandit models [3, 28], and adversarial models [10], as well.

Applications of our model include the following examples:

Example 1 (Matching constraint) . We consider the problem of assigning L jobs to K workers. Let V 1 and V 2 be the sets of workers and jobs, respectively. Let E ⊆ V 1 × V 2 be a set of pairs of a worker and a job such that ( u, v ) ∈ E if and only if the worker u is qualified to do the job v . In each round, we choose a subset M of E so that no two elements in M share the same worker or job, that is, M must satisfy the matching constraint for the bipartite graph G = ( V 1 ∪ V 2 , E ) . Each job has processing time to complete, and, when a job is assigned to a worker, the job-worker pair is occupied until the job is completed. Thus the occupied job-worker pairs have to be included in M . We assume that the time required to complete a job and the quality of the results (rewards) follow unknown distributions that depend on the job-worker pair. This problem can be regarded as a special case of the bandit task assignment problem with N = | E | . Choosing a t ∈ A corresponds to assigning a job v to a worker u for each e = ( u, v ) such that a te = 1 , and the worker u and the job v will then be occupied and unavailable until the task is completed. It would be worth mentioning that the matching constraint is often the subject of consideration in the context of combinatorial semi-bandits [18, 30].

Example 2 (Knapsack constraint) . Consider the situation where we want to perform tasks (e.g., of data processing) under resource constraints (e.g., computational resources such as CPUs, GPUs, and memories). We assume that the processing time of each task varies stochastically, and is not revealed until it is completed. Formally, we are given N tasks and K resources. Assume that each task i ∈ [ N ] consumes b ki ∈ R ≥ 0 amount of resource k ∈ [ K ] per unit time, and that the total

amount of resource k available in unit time is f k . Then the player can choose a set of tasks from A = { a ∈ { 0 , 1 } N | b glyph[latticetop] k a ≤ f k ( k ∈ [ K ]) } , where b k = ( b k 1 , b k 2 , . . . , b kN ) glyph[latticetop] ∈ R N ≥ 0 for each resource k . Note that the set of all running tasks must satisfy the constraint given by A , which means that we cannot use resources occupied by tasks in progress and we are restricted to choose a set of tasks that can be executed with remaining resources. We note that the class of problems with a single resource ( K = 1 ) includes the budget-constrained multi-armed bandit [20, 15] as a special case.

Example 3 (Matroid constraint [35, 17]) . If A consists of indicator vectors for independent sets of a matroid with the ground set [ N ] , we call this a matroid constraint. Matroid constraints capture a variety of problem settings of practical importance, including the situation in which the number of tasks that can be executed simultaneously is limited to at most K , i.e., A = { a ∈ { 0 , 1 } N | ‖ a ‖ 1 ≤ K } . This special case is called a uniform matroid.

1.1 Contribution

Our main contribution is to present an algorithm that achieves a regret bound of O ( C u C 2 l MN (1 / ∆+ C u ) ln T ) , where C u and C l are upper and lower bounds for the processing times c ti , M is the maximum of ‖ a ‖ 1 for a ∈ A , and ∆ > 0 is the gap between expected per-round rewards for the optimal and the best suboptimal actions. The proposed algorithm also enjoys a gap-free regret bound of O ( 1 C l √ C u MNT ln T ) . Our contributions include a nearly tight lower bound as well, i.e., we show that any algorithm suffers a regret at least Ω( 1 C l √ C u MNT ) . This indicates that the proposed algorithm achieves nearly optimal regret bounds.

To design our algorithm, we first show that a nearly optimal policy can be characterized by expected per-round rewards for tasks (Proposition 2.2 in Section 2). This implies that, given estimation for expected per-round rewards and a linear optimization oracle for A , we can compute asymptotically optimal actions. This is in contrast to the blocking bandits, in which it is computationally hard to maximize the total reward even if all the distributions are known, as presented, e.g., in Corollary 3.2 by Basu et al. [8]. That is why, for the blocking bandits, existing algorithms can achieve only approximate regret bounds, i.e., the performance is only guaranteed to be (close to) a constant-factor approximation for the optimal strategy.

Proposition 2.2 suggests us to balance exploration and exploitation with the aid of upper confidence bounds (UCB) on the expected per-round reward for each task, which can be computed efficiently. Such a UCB-based approach is similar to CombUCB1 for combinatorial semi-bandits [25]. CombUCB1 , however, does not directly apply to our problem setting due to the stochastically variable processing time. In the bandit task assignment problem, the set of tasks that the player can start in each round changes depending on the set of tasks still in progress, which makes the decision of the player restricted. To address this issue, we introduce a phased-update approach, in which we divide the entirety of the rounds [ T ] = { 1 , 2 , . . . , T } into segments of appropriate length. At the beginning of each segment s , we compute a set of tasks A ′ s based on the UCB for expected per-round rewards. We note that this can be done independently of the state of running tasks. Then, in the s -th segment, we basically continue to perform the same set of tasks A ′ s .

One main technical difficulty is how to determine the length l s of the segment s . At the start of a new segment s , tasks in A ′ s cannot be executed until all tasks performed in the previous segment have been completed, i.e., every switching point between segments causes idle time with non-zero probability. Therefore, the larger the number of segments (i.e., the shorter the length of each segment), the greater the loss due to such waiting time. In particular, if the segment length is O ( C ) , we cannot avoid a linear regret of Ω( T/C ) due to idle time. On the other hand, setting too long segments also has a negative impact on performance as the longer the length of each segment, the less frequently the set A ′ s will be updated, which may degrade the efficiency of exploration and exploitation. To address this dilemma, this study designs a way of determining the segment length l s by which these potential losses will be well balanced and the overall regret will be bounded as desired.

Another nontrivial technique in the proposed algorithm is to employ Bernstein-type confidence bounds [4, 27] for the processing time. Thanks to this, we can construct a tight UCB estimator for the expected per-round reward for each task i , with a width depending on the mean ¯ c i and variance σ 2 i of the processing time. This is essential to achieve the nearly optimal ˜ O ( 1 C l √ C u MNT ) -regret bound. In fact, as we mention in Remark 4.2, an algorithm with standard confidence bounds will lead to regret upper bounds with additional C u /C l factors, which do not match the lower bound.

Summary of contributions The contributions of this paper are summarized in the following four points: (i) We propose a novel problem setting of the bandit task assignment, which incorporates simultaneously combinatorial constraints among tasks, (unknown) processing time of each task, and the exploration-exploitation trade-off. The model includes various practical applications as described in Examples 1, 2 and 3. (ii) We show that a nearly optimal policy can be characterized with expected per-round rewards, which can be computed efficiently given only a linear optimization oracle. This contrasts with the computational difficulty in blocking bandits. (iii) We provide an algorithm with nearly optimal regret bounds. We handle the difficulties arising from the combinatorial constraints together with processing times by the phased-update approach. (iv) We present a regret lower bound that matches to the regret bound of the proposed algorithm, ignoring logarithmic factors. This means that the proposed algorithm achieves nearly optimal performance in terms of the worst-case analysis.

2 Problem Setup

Let N be the number of tasks. The player is given a family of feasible sets of tasks, which is expressed by A ⊆ { 0 , 1 } N . Here, each binary vector a in { 0 , 1 } N corresponds to a set of tasks, and a ∈ A means that the set of tasks A = { i ∈ [ N ] | a i = 1 } can be executed simultaneously. We assume that A is closed under inclusion, i.e., a ∈ A and b ≤ a together imply b ∈ A . Note here that, for any vectors a = ( a 1 , . . . , a N ) glyph[latticetop] , b = ( b 1 , . . . , b N ) glyph[latticetop] ∈ R N , the notation of b ≤ a means that b i ≤ a i holds for all i ∈ { 1 , . . . , N } . We denote M = max a ∈A ‖ a ‖ 1 . In each round t , the player will choose a set of tasks a t from A . The chosen set has to exclude all the processing tasks not yet completed. More precisely, a t is constrained to satisfy a t + b t ∈ A , where b t is the set of tasks still in progress at the t -th round (see (1) below). Each task i with a ti = 1 is then started and will be completed at the beginning of the ( t + c ti ) -th round, which yields the reward of r ti . The reward r ti and the processing time c ti will be revealed only after the task is completed. We assume that r ti and c ti are restricted so that r ti ∈ [0 , 1] and c ti ∈ { C l , C l +1 , . . . , C u } , where C l and C u are integers satisfying 1 ≤ C l ≤ C u . We also assume that (( r ti , c ti )) N i =1 follows an unknown distribution D over ([0 , 1] ×{ C l , . . . , C u } ) N , independently for t = 1 , 2 , . . . , T . Note that r ti and c ti may be dependent. From the above problem definition, a family A t of task sets available at the t -th round is expressed as follows:

where the vector b t = ( b ti ) N i =1 corresponds to the set of uncompleted tasks that starts before the t -th round. The goal of the player is to maximize the sum of rewards earned by the T -th round, which can be expressed as ∑ T t =1 r glyph[latticetop] t a t .

Remark 2.1 . Stochastic combinatorial semi-bandits [25] and the multi-armed bandit problem with a budget constraint [15] are special cases of the bandit task assignment problem. In fact, if c ti = 1 for all t and i , the problem corresponds to a combinatorial semi-bandit problem with action set A . In addition, if A = { a ∈ { 0 , 1 } N | ‖ a ‖ 1 ≤ 1 } , the problem corresponds to an N -armed bandit with budget B = T and costs { c ti } .

In the problem, we assume that we are able to solve linear optimization over A efficiently. More precisely, we suppose that we are given access to an offline linear optimization oracle that returns a solution ˆ a ∈ arg max a ∈A q glyph[latticetop] a for any input q ∈ R N ≥ 0 . Note here that, for the problems with matching constraints or with matroid constraints (Examples 1 and 3, respectively), such an oracle can be implemented in polynomial time [23, 17]. For Example 2, linear optimization over A is NP-hard, but can be solved practically fast in many cases, with the aid of dynamic programming or integer programming algorithms.

We define the regret R T to be the expectation of the difference between the total rewards obtained with the optimal proper policy and those obtained with the algorithm by the T -th round, where we call a policy for choosing a t proper if a t satisfies the constraint defined by (1) and each a t depends on information observed by the beginning of t -th round, but not on (( r s , c s )) T s = t . Any proper policy can be expressed as sequences of mapping π = ( π t ) t =1 , 2 ,... , where each π t maps from the history h t , which consists of chosen actions ( a s ) t -1 s =1 and feedback obtained by the beginning of the t -round, to

action a t in A . Let Π denote the set of all proper policies. The regret can then be expressed as

where ( h ∗ t ) t =1 , 2 ,... denotes histories generated by the policy π ∗ . Intuitively, the total reward obtained with the optimal proper policy corresponds to the maximum performance achieved by an agent with unlimited computational resources who knows the distribution D of processing times and rewards. Note that the regret R T is defined on the basis of tasks started by the T -th round. Although it may seem more natural to define the regret by the rewards for tasks completed by the T -th round, the difference between these two alternatives is only O ( M/C l ) , which is almost negligible as it is independent of T .

We can see that the performance of the optimal proper policy (and therefore also regret) can be characterized by using the expected per-round rewards. We define ¯ r i and ¯ c i to be the expectation of r ti and c ti , respectively. We also denote q i = ¯ r i / ¯ c i . Define vectors ¯ r ∈ [0 , 1] N , ¯ c ∈ [ C l , C u ] N and q ∈ [0 , 1 /C l ] N by ¯ r = (¯ r 1 , ¯ r 2 , . . . , ¯ r N ) glyph[latticetop] , ¯ c = (¯ c 1 , ¯ c 2 , . . . , ¯ c N ) glyph[latticetop] and q = ( q 1 , q 2 , . . . , q N ) glyph[latticetop] = (¯ r 1 / ¯ c 1 , ¯ r 2 / ¯ c 2 , . . . , ¯ r N / ¯ c N ) glyph[latticetop] . Note that q i corresponds to the expected per-round reward for running each task i . Then, the expected total reward for an optimal policy can be bounded as follows:

Proposition 2.2 can be considered as a generalization of Lemma 1 by Ding et al. [15]. From this proposition, the regret can be bounded as follows:

where we denote a ∗ ∈ arg max a ∈A q glyph[latticetop] a and the last inequality follows from the facts that q i ≤ 1 /C l holds for any i and that ‖ a ∗ ‖ 1 ≤ M .

3 Algorithm

The proposed algorithm employs a phased-update approach, in which we update a policy for each phase, and each phase continues to select the same set of tasks.

The procedure starts with the initialization phase (or the zeroth phase), in which we execute each task B = Θ( C u C l ln T ) times. This is required to ensure that the UCB estimators ˆ q i ( t ) given in (3) are close enough to the true expected reward q i . Let t 1 ≥ 1 denote the round this initialization terminates, which is at most O ( C u NB ) . As the regret per round is at most M C l , the total regret in the initialization phase is at most O ( C u MNB C l ) = O ( C 2 u C 2 l MN ln T ) , which turns out to be negligible in the overall regret upper bound.

The s -th phase ( s ∈ { 1 , 2 , . . . } ) consists of a segment of rounds with length l s , which implies that the first round t s of the s -th phase is expressed as t s = ∑ s -1 u =1 l u + t 1 . At the beginning of each phase s , we compute a ′ s ∈ A and the length l s of the s -th phase, for which the details will be described later. Then, in the subsequent l s rounds, we continue to choose the set A ′ s = { i ∈ [ N ] | a ′ si = 1 } . More precisely, in the t -th round, we choose a t = a ′ s -b t if b t ≤ a ′ s and choose a t = 0 otherwise, where we recall b t is the set of tasks in progress at the t -th round (see (1)). Since a ′ s ∈ A , a t clearly belongs to A t . In this procedure, intuitively, we repeatedly restart each task i in A ′ s just after i is completed, in the s -th phase. Note here that all tasks executed in the ( s -1) -st phase are completed by the ( t s + C u ) -th round as the processing times are at most C u , which implies that b t ≤ a ′ s holds for any t ∈ [ t s + C u , t s +1 -1] . Hence, in any round t ∈ [ t s + C u , t s +1 -1] , it holds that b t + a t = a ′ s , which implies all tasks in A ′ s are running in the phase. Furthermore, for each task i ∈ A ′ s , the number of times the task i is completed in the s -th phase will be at least glyph[floorleft] ( l s -C u ) /C u glyph[floorright] ( ≥ l s /C u -2) and at most l s /C l +1 .

We next describe how to compute a ′ s . We use upper confidence bounds (UCB) for q i = ¯ r i / ¯ c i , the expected reward per round. In our definition of the UCB, we use the empirical means ˆ r i ( t ) and ˆ c i ( t )

of rewards and costs, the empirical variance V c i ( t ) of costs, and the number T i ( t ) of times that the task i has been completed before the t -th round. We define the UCB ˆ q i ( t ) on q i by

Wewould like to note this definition includes an empirical Bernstein bound d c i ( t ) [4] for the processing time c ti , which will turn out to be essential for tight regret upper bounds, as mentioned in Remark 4.2. We then have

These can be shown from Azuma-Hoeffding inequality and empirical Bernstein's inequality [4], of which details can be found in Section A.4 of the appendix. Hence, with probability 1 -6 /t 2 , we have q i ≤ ˆ q i ( t ) . At the beginning of each phase s , i.e., in the t s -th round, we calculate these UCBs ˆ q ( t s ) = (ˆ q 1 ( t s ) , ˆ q 2 ( t s ) , . . . , ˆ q N ( t s )) glyph[latticetop] , and then call the linear optimization oracle to find a ′ s ∈ arg max a ∈A ˆ q ( t s ) glyph[latticetop] a .

We set the length l s of the s -th phase on the basis of min i ∈ A ′ s T i ( t s ) , where we denote A ′ s = { i ∈ [ N ] | a ′ si = 1 } . We set l s = C l min i ∈ A ′ s T i ( t s ) + 2 C u . In the analysis, we assume that B is given by B = glyph[ceilingleft] 90 C u C l ln T glyph[ceilingright] . We then have

which will be used at several points in our analysis. These conditions and the construction of l s lead to the bound on T i ( t s ) as in the lemma below, which will be used in the regret analysis for the proposed algorithm.

Lemma 3.1. For any s ≥ 1 and i ∈ A ′ s , we have l s ≤ 2 C u ( T i ( t s +1 ) -T i ( t s )) and T i ( t s +1 ) ≤ 4 T i ( t s ) .

All omitted proofs are given in the appendix. We also have the following bound on t s :

Lemma 3.2. For any s ≥ 1 , there exists i ∈ A ′ s such that T i ( t s +1 ) ≥ (1 + C l /C u ) T i ( t s ) . Consequently, we have t s ≥ C l (1 + C l /C u ) s/N -2 for any s ≥ 1 .

From Lemma 3.2, we have ln t s ≥ ln C l + ( s N -2 ) ln ( 1 + C l C u ) ≥ ln C l + ( s N -2 ) C l 2 C u , where the last inequality follows from the fact that ln(1 + x ) ≥ x/ 2 for any x ∈ [0 , 1] . This implies that we have s ≤ N ( 2 C u C l ln t s +2 ) . Hence, the number of phases, denoted by S , is bounded by S ≤ N ( 2 C u C l ln T +2 ) +1 = O ( C u C l N ln T ) , since the last phase contains T .

The overall procedure of the proposed algorithm is summarized in Algorithm 1, which consists of arithmetic operations and offline linear optimization over A . The number of arithmetic operations in each round is bounded by O ( N ) since the update of each parameter in Step 7-10 of Algorithm 1 can be performed in O (1) -arithmetic operations. Furthermore the number of calls to the offline linear optimization oracle is at most S = O ( C u C l N ln T ) . In fact, the offline linear optimization oracle is called only at the beginning of each phase. This means that Algorithm 1 is more efficient than standard UCB algorithms for combinatorial semi-bandits algorithms [24, 25] that require O ( T ) calls to the oracle. The space complexity, other than that required for the offline optimization oracle, is O ( N ) since the algorithm works by maintaining O ( N ) parameters.

4 Regret Analysis

We denote a ∗ ∈ arg max a ∈A q glyph[latticetop] a . Let A ∗ = { i ∈ [ N ] | a ∗ i = 1 } and ˜ A = [ N ] \ A ∗ . We define suboptimality gaps ∆ a , ∆ i and ∆ min , respectively, by

Algorithm 1 Phased-update UCB algorithm for bandit task assignment

Require: A ⊆ { 0 , 1 } N : a family of feasible task sets. C l , C u : lower and upper bounds for the processing time. B : length of initialization phase. Offline linear optimization oracle for A .

1: For each i ∈ [ N ] , execute the i 's task B times, and let t 1 be the round it completed. Set T i ( t 1 ) = B .

2: for s = 1 , 2 , . . . , do

3: Set ˆ q i ( t s ) by (3).

4: Call the offline linear optimization oracle to find a ′ s ∈ arg max a ∈A ˆ q ( t s ) glyph[latticetop] a , and set A ′ s = { i ∈ [ N ] | a ′ si = 1 } .

5: Set l s = C l min i ∈ A ′ s T i ( t s ) + 2 C u and t s +1 = t s + l s .

7: If b t ≤ a ′ s , output a t = a ′ s -b t , where b t is given by (1). Otherwise, output a t = 0 .

6: for t = t s , t s +1 , . . . , t s +1 -1 do

8: for i = 1 , 2 , . . . , N do

9: If task i is completed (i.e., b ti = 0 ) and c t ′ ,i and r t ′ ,i are observed ( t ′ is the last round the task i is started), update ¯ c i ( t ) , ¯ r i ( t ) and V c i ( t ) by using c t ′ ,i and r t ′ ,i . Set T i ( t +1) = T i ( t ) + 1 .

10: end for

11: end for

12: end for

We also denote the variance of the processing time c ti by σ 2 i = E [( c ti -¯ c i ) 2 ] . We note that σ 2 i ≤ E [( c ti -C l ) 2 ] ≤ ( C u -C l ) E [ c ti -C l ] = ( C u -C l )(¯ c i -C l ) .

The goal of this section is to show the following:

Theorem 4.1. The regret for Algorithm 1 is bounded as

where ˜ C i ≤ O ( 1 ¯ c i ( 1 + 1 ¯ c 2 i ( σ 2 i + ( C u -C l ) 2 C l C u ))) ≤ O ( 1 ¯ c i ( 1 + C u -C l ¯ c i )) ≤ O ( C u ¯ c i C l ) . Furthermore, for any distribution D , the regret is bounded as R T = O ( 1 C l √ C u NMT ln T + C 2 u C 2 l NM ln T ) .

The specific definition of ˜ C i is given in (19) in the appendix.

Remark 4.2 . Bernstein-type confidence bounds used in (3) are essential to achieve the nearly optimal regret bound of R T = O ( 1 C l √ C u NMT ln T ) . In fact, if we employ a standard confidence bound for ¯ c i instead of the Bernstein-type one given in (3), the parameter σ 2 i in the definition of ˜ C i will be replaced with ( C u -C l ) 2 , which leads a suboptimal regret bound of R T = O (√ C 2 u C 3 l T ln T ) .

From Proposition 2.2, the regret can be bounded as follows:

where we define

In these definitions of R (1) T and R (2) T , the index S represents the phase that includes the T -th round, and we define t S +1 = T +1 exceptionally, for notational simplicity.

Let us next show the upper bounds for R (1) T and R (2) T separately. We will show that

These, together with (8) and (5), prove the gap-dependent regret bound (7) in Theorem 4.1.

We here discuss an upper bound on R (1) T while the analysis of R (2) T is given in the supplementary material. We denote ˜ A s = A ′ s \ A ∗ . Define d i ( t ) by d i ( t ) = √ ˜ C i ¯ c i ln t T i ( t ) , where ˜ C i is defined by (19) in the appendix so that ˜ C i = Θ ( 1 ¯ c i ( 1 + 1 ¯ c 2 i ( σ 2 i + ( C u -C l ) 2 C l C u ))) . We then have ˆ q i ( t ) -q i ( t ) ≤ d i ( t ) with high probability. In fact, we can show the following lemma using concentration inequalities, of which proof can be found in the appendix.

Lemma 4.3. For any s , with a probability at least 1 -6 N/t 2 s , we have ∆ a ′ s ≤ ∑ i ∈ ˜ A s d i ( t s ) . Consequently, we have R (1) T ≤ E [ ˆ R T ] + O ( MN C l t 1 ) , where we define ˆ R T = ∑ S s =1 l s ∆ a ′ s 1 [ F s ] with F s = { ∆ a ′ s ≤ ∑ i ∈ ˜ A s d i ( t s ) , ∆ a ′ s > 0 } .

We can show an upper bound on ˆ R T in a way similar to what Kveton et al. [25] did, as follows:

This bound on ˆ R T together with Lemma 4.3 and (5) leads to the first part of (10).

4.1 Gap-Free Regret Bound

We can obtain the gap-free regret bound of R T = ˜ O ( √ C u C 2 l NMT ) in Theorem 4.1 by modifying the analysis of ˆ R T . For any glyph[epsilon1] > 0 , ˆ R T can be bounded as ˆ R T ≤ ∑ S s =1 ( l s ∆ a ′ s 1 [ F s , ∆ a ′ s > glyph[epsilon1] ] + l s ∆ a ′ s 1 [ F s , ∆ a ′ s ≤ glyph[epsilon1] ] ) = O ( C u C 2 l MN ln T glyph[epsilon1] + glyph[epsilon1]T ) , where the last inequality follows from the same argument for showing Lemma 4.4. By setting glyph[epsilon1] = √ C u MN ln T C 2 l T , we obtain ˆ R T = O ( 1 C l √ C u MNT ln T ) . From this, (8), the second part of (10), and Lemma 4.3, we obtain the gap-free regret bound presented in Theorem 4.1. This completes the proof of Theorem 4.1.

4.2 Regret Lower Bound

As shown in Theorem 4.1, the proposed algorithm achieves R T = O ( 1 C l √ C u NMT ) . This is tight up to a logarithmic factor in T . In fact, we have the following lower bound:

Theorem 4.5. For any N,M,T such that N ≥ M and T = Ω( C u ) , there exists a problem instance for which any algorithm suffers regret of R T = Ω( 1 C l min { √ C u NMT,MT } ) .

In the proof of this theorem, we first focus on the case of M = 1 . If we consider the case of c ti = C l , our model is equivalent to the N -armed bandit with time horizon T ′ = Θ( T/C l ) , which leads to a lower bound of Ω( √ NT ′ ) = Ω( √ NT/C l ) with the aid of the lower bound for multi-armed bandit problems given in Theorem 5.1 by Auer et al. [6]. When considering the case of r ti = 1 and c ti ∈ { C l , C u } , we can show a regret lower bound of Ω( C u -C l C l C u √ C u NT ) , by using the proof technique by Badanidiyuru et al. [7]. Combining these two bounds, we obtain an Ω( 1 C l √ C u NT ) -lower bound for the case of M = 1 . From this result and the technique used in Proposition 2 by Kveton et al. [25], we have an Ω( 1 C l √ C u MNT ) -lower bound for M > 1 .

5 Numerical Experiment

We conducted small experiments to evaluate the practical performance of the proposed algorithm using a synthetic dataset. We set the parameters for the dataset as follows: A = { A ⊆ [ N ] |

0

2000

4000

6000

8000

10000

round

0

500

1000

1500

2000

regret

UCB-BV1

CombUCB1

Proposed

0

2000

4000

6000

8000

10000

round

0

500

1000

1500

2000

2500

3000

3500

regret

UCB-BV1

CombUCB1

Proposed

| A | ≤ M } with ( N,M ) = (4 , 2) , T = 10000 , C u = 6 , C l = 1 , ¯ r = (0 . 5 , 0 . 5 , 0 . 5 , 0 . 5) . For the expected processing times, we set ¯ c = (1 . 5 , 1 . 5 , 2 . 0 , 2 . 0) as a problem instance with small ∆ or ¯ c = (1 . 5 , 1 . 5 , 5 . 0 , 5 . 0) as a problem instance with large ∆ . Each c ti and r ti are generated so that c ti -1 follows binomial distributions over { 0 , 1 , . . . , C u -1 } and r ti follows Bernoulli distributions over { 0 , 1 } . For comparison, we have added the results of applying UCB-BV1 by Ding et al. [15] and CombUCB1 by Kveton et al. [24]. More detailed information on the experimental setup is provided in the supplementary material.

Figures 2 and 3 show the empirical means of regret computed by 100 repetitions of independent experiments. We depict 2 -standard deviations of the empirical regret by the shaded areas. These results imply that the proposed algorithm performs better experimentally than other algorithms. We note that UCB-BV1 and CombUCB1 cannot avoid a linear regret as they choose a set in A and wait for them all to be completed, which causes idling time with non-zero probability every time. It is also worth mentioning that the empirical performance of the proposed algorithm is much better than Theorem 4.1 predicts. In fact, under the parameter settings of these experiments, the values in Theorem 4.1 can be approximated as: 1 C l √ C u NMT ln T ≈ 2100 and C u C l NM ln T ≈ 2650 .

6 Related Work

One of the most relevant studies to this work would be those on combinatorial semi-bandits , in which a family F ⊆ 2 [ N ] of subsets of arms [ N ] is given and the player sequentially chooses A t ∈ F and then observes the obtained rewards r ti for each i ∈ A t . Studies on combinatorial semi-bandits are classified into those on stochastic models [25, 24, 34], in which r t are i.i.d. for t , and those on adversarial models [33, 5], in which { r t } are arbitrary sequences that may be chosen in an adversarial manner. As noted in the introduction, the stochastic combinatorial semi-bandits are special cases of bandit task assignment. The stochastic combinatorial semi-bandits are known to admit sublinear regret. Typical approaches to achieve sublinear regret include upper confidence bounds (UCB)-type algorithms [25, 24] and Thompson sampling algorithms [34].

Our problem is also relevant to bandit problems with budget constraints. Ding et al. [15] consider a stochastic multi-armed bandit (MAB) problem with a budget constraint, in which the player observes the reward r ti t and the cost c ti t for the chosen arm i t in each round t . The goal of the player is to maximize the cumulative rewards earned before cumulative costs reach a given total budget amount. This can be seen as a special case of our problem. As shown in Ding et al. [15], in the budget-constrained MAB problem, the reward of the optimal policy can be characterized by the expected reward-to-cost. This fact is generalized to our setting in Proposition 2.2, which will be used in our regret analysis. It should be mentioned that the budget-constrained MAB problem has been generalized to bandits with multiple plays [37], which differs from our model, as problems with processing times cannot be reduced to budget-constrained models in general. In addition to their work, there are various generalizations and variants, such as contextual bandit models [36], models with general distributions [11, 12], and models with multiple budgets [2, 31, 7].

While delayed feedback models [13, 38, 3] are similar to our model in that rewards earned will be revealed in later rounds, there are some essential differences. For example, in our model, the selected arm is occupied during the processing time and cannot be selected, while in the delayed feedback model, the arm can be selected while waiting for feedback. The former model can handle situations

where the arm corresponds to a labor or computing resource, as in the examples shown in Examples 1 and 2.

Phased-update or lazy-update approaches were incorporated in other bandit problems and online learning problems such as a phased version of UCB for MAB (e.g., Exercise 7.5 of the book [26]) and the rarely switching OFUL algorithm for linear stochastic bandits [1]. The motivation for using phased-update approaches is primarily to reduce the computational cost [1, 22], to minimize switching cost [2, 21], or to address batched settings [29, 19]. To our knowledge, our proposed algorithm is the first algorithm that applies such a phased-update approach to combinatorial semi-bandits. It allows us to reduce the number of oracle calls, and more importantly, to achieve the nearly tight regret bounds for the bandit task assignment. Our study reveals new usefulness of a phased-update approach.

Another line of research related to our problem is the online matching or online allocation problems with reusable resources [14, 16, 32]. The problem is formalized with a bipartite graph, and we choose edges for sequentially arriving vertices. The model is similar to ours in that each chosen edge will be unavailable for a certain period of time that follows a probability distribution. However, it is different in that the bipartite graph is unknown at first and all the distributions are given. Moreover, those papers adopt competitive ratio analyses rather than regret analyses, i.e., they evaluate the performance compared to a stronger agent who knows the sequence of arriving vertices. Thus, the existing papers are incomparable with our work.

As mentioned in the introduction, the problem setting of blocking bandit [8] appears to be similar to ours while the difficulties are significantly different. Considering unconstrained situations can highlight the differences in the problem. In the task assignment problem with no constraints, the optimal strategy will be trivial, in which we make each task always processing, i.e., in each round, we start every task that is completed at that round. On the other hand, as the reviewer commented, blocking bandits only allow us to play a single action in each round, which makes the optimal strategy nontrivial. In fact, finding the optimal policy can be an NP-hard problem even if the true distributions of r and c are given. This is an illustrative example of why bandit task assignment can be easier than blocking bandits.

7 Conclusion and limitation

In this paper, we introduced a new problem setting that we referred to as bandit task assignment, for which we proposed an algorithm with regret bounds. A limitation of this work is that we need to know the upper and lower bounds C u , C l on the processing times, which may not be possible to assume in some real-world applications. To remove this assumption, it would be necessary to modify the framework of the problem. One possible modification would be to allow the player to suspend processing tasks. In this modified problem setup, even in cases where C u is incorrectly underestimated, it will be possible to immediately abort the task and correct C u when the underestimation is detected. We expect that this strategy and the appropriate parameter update rules for C u and C l allow us to remove the assumption of prior knowledge about processing times. Another future direction is to extend the problem to a decentralized setting, in which multiple agents collaborate to make task assignment decisions. We believe that such an extension of the problem provides insight into real-world applications where multiple entities perform task assignments, such as crowdsourcing and federated learning.

Acknowledgments

Wedeeply appreciate many comprehensive comments and feedback by anonymous reviewers. HS was supported by JSPS KAKENHI Grant Numbers JP17K12646, JP21K17708, and JP21H03397, Japan. TF was supported by JSPS KAKENHI Grant Numbers JP20H05965, JP21K11759, and JP21H03397, Japan. NK was supported by JSPS KAKENHI Grant Numbers JP22H05001, JP20H05795, and JP21H03397, Japan. KK was supported by JSPS KAKENHI Grant Numbers JP22H05001 and JP20A402, Japan.

References

Y. Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems , pages 2312-2320, 2011.

R. Agrawal, M. Hegde, D. Teneketzis, et al. Multi-armed bandit problems with multiple plays and switching cost. Stochastics and Stochastic reports , 29(4):437-459, 1990.

A. Atsidakou, O. Papadigenopoulos, S. Basu, C. Caramanis, and S. Shakkottai. Combinatorial blocking bandits with stochastic delays. In International Conference on Machine Learning , pages 404-413. PMLR, 2021.

J.-Y. Audibert, R. Munos, and C. Szepesvári. Tuning bandit algorithms in stochastic environments. In International conference on algorithmic learning theory , pages 150-165. Springer, 2007.

J.-Y. Audibert, S. Bubeck, and G. Lugosi. Regret in online combinatorial optimization. Mathematics of Operations Research , 39(1):31-45, 2013.

P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing , 32(1):48-77, 2002.

A. Badanidiyuru, R. Kleinberg, and A. Slivkins. Bandits with knapsacks. Journal of the ACM (JACM) , 65(3):1-55, 2018.

S. Basu, R. Sen, S. Sanghavi, and S. Shakkottai. Blocking bandits. Advances in Neural Information Processing Systems , 32:4785-4794, 2019.

S. Basu, O. Papadigenopoulos, C. Caramanis, and S. Shakkottai. Contextual blocking bandits. In International Conference on Artificial Intelligence and Statistics , pages 271-279. PMLR, 2021.

N. Bishop, H. Chan, D. Mandal, and L. Tran-Thanh. Adversarial blocking bandits. Advances in Neural Information Processing Systems , 33:8139-8149, 2020.

S. Cayci, A. Eryilmaz, and R. Srikant. Learning to control renewal processes with bandit feedback. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 3(2): 1-32, 2019.

S. Cayci, A. Eryilmaz, and R. Srikant. Budget-constrained bandits over general cost and reward distributions. In International Conference on Artificial Intelligence and Statistics , pages 4388-4398. PMLR, 2020.

N. Cesa-Bianchi, C. Gentile, and Y. Mansour. Delay and cooperation in nonstochastic bandits. Journal of Machine Learning Research , 20(17):1-38, 2019.

J. P. Dickerson, K. A. Sankararaman, A. Srinivasan, and P. Xu. Allocation problems in ridesharing platforms: Online matching with offline reusable resources. ACM Transactions on Economics and Computation , 9(3), June 2021.

W. Ding, T. Qin, X.-D. Zhang, and T.-Y. Liu. Multi-armed bandit with budget constraint and variable costs. In Twenty-Seventh AAAI Conference on Artificial Intelligence , 2013.

Z. Dong, S. Das, P. Fowler, and C.-J. Ho. Efficient nonmyopic online allocation of scarce reusable resources. In International Conference on Autonomous Agents and Multiagent Systems , pages 447-455, 2021.

J. Edmonds. Matroids and the greedy algorithm. Mathematical programming , 1(1):127-136, 1971.

Y. Gai, B. Krishnamachari, and R. Jain. Learning multiuser channel allocations in cognitive radio networks: A combinatorial multi-armed bandit formulation. In 2010 IEEE Symposium on New Frontiers in Dynamic Spectrum (DySPAN) , pages 1-9. IEEE, 2010.

Z. Gao, Y. Han, Z. Ren, and Z. Zhou. Batched multi-armed bandits problem. Advances in Neural Information Processing Systems , 32:501-511, 2019.

A. György, L. Kocsis, I. Szabó, and C. Szepesvári. Continuous time associative bandit problems. In Proceedings of the 20th international joint conference on Artifical intelligence , pages 830835, 2007.

T. Jun. A survey on the bandit problem with switching costs. de Economist , 152(4):513-541, 2004.

A. Kalai and S. Vempala. Efficient algorithms for online decision problems. Journal of Computer and System Sciences , 71(3):291-307, 2005.

H. W. Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83-97, 1955.

B. Kveton, Z. Wen, A. Ashkan, H. Eydgahi, and B. Eriksson. Matroid bandits: fast combinatorial optimization with learning. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence , pages 420-429, 2014.

B. Kveton, Z. Wen, A. Ashkan, and C. Szepesvari. Tight regret bounds for stochastic combinatorial semi-bandits. In International Conference on Artificial Intelligence and Statistics , pages 535-543. PMLR, 2015.

T. Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

A. Maurer and M. Pontil. Empirical bernstein bounds and sample variance penalization. arXiv preprint arXiv:0907.3740 , 2009.

O. Papadigenopoulos and C. Caramanis. Recurrent submodular welfare and matroid blocking semi-bandits. Advances in Neural Information Processing Systems , 34:23334-23346, 2021.

V. Perchet, P. Rigollet, S. Chassang, and E. Snowberg. Batched bandit problems. The Annals of Statistics , 44(2):660-681, 2016.

P. Perrault, E. Boursier, M. Valko, and V. Perchet. Statistical efficiency of thompson sampling for combinatorial semi-bandits. Advances in Neural Information Processing Systems , 33: 5429-5440, 2020.

K. A. Sankararaman and A. Slivkins. Combinatorial semi-bandits with knapsacks. In International Conference on Artificial Intelligence and Statistics , pages 1760-1770. PMLR, 2018.

H. Sumita, S. Ito, K. Takemura, D. Hatano, T. Fukunaga, N. Kakimura, and K. Kawarabayashi. Online task assignment problems with reusable resources. In The Thirty-Sixth AAAI Conference on Artificial Intelligence , 2022.

T. Uchiya, A. Nakamura, and M. Kudo. Algorithms for adversarial bandit problems with multiple plays. In International Conference on Algorithmic Learning Theory , pages 375-389, 2010.

S. Wang and W. Chen. Thompson sampling for combinatorial semi-bandits. In International Conference on Machine Learning , pages 5114-5122. PMLR, 2018.

H. Whitney. On the abstract properties of linear dependence. American Journal of Mathematics , 57(3):509-533, 1935.

H. Wu, R. Srikant, X. Liu, and C. Jiang. Algorithms with logarithmic or sublinear regret for constrained contextual bandits. Advances in Neural Information Processing Systems , 28: 433-441, 2015.

D. P. Zhou and C. J. Tomlin. Budget-constrained multi-armed bandits with multiple plays. In Thirty-Second AAAI Conference on Artificial Intelligence , pages 4572-4579, 2018.

J. Zimmert and Y. Seldin. An optimal algorithm for adversarial bandits with arbitrary delays. In International Conference on Artificial Intelligence and Statistics , pages 3285-3294. PMLR, 2020.

A Omitted Proofs

Lemma A.1 (Empirical bernstein inequalities, [4]) . Let X 1 , X 2 , . . . , X n ∈ [ C l , C u ] be a sequence of i.i.d. random variables with mean µ and variance σ 2 . Let ˆ µ n = ∑ n i =1 X i /n and V n = ∑ n i =1 ( X i -ˆ µ n ) 2 /n . For any δ ∈ (0 , 1) , with probability 1 -4 δ , we have

Proof. Theorem 1 by Audibert et al. [4] implies that the first inequality holds with probability 1 -3 δ . The second inequality follows from Bernstein's inequality for V t :

From this, with probability 1 -δ , we have

where the second inequality follows from √ 1 + x + y ≤ √ 1 + x + √ y ≤ 1 + x/ 2 + √ y that holds for any x, y ≥ 0 . This implies that the second inequality of (11) holds with probability 1 -δ .

A.1 Proof of Proposition 2.2

Proof. For t = 1 , 2 , . . . , T , denote a t = π t ( h t ) and define b t by (1). For t > T , let a t = 0 for the notational simplicity. Then, as r t is independent of a t , we have

From (1), we have E [ ∑ T + C u t =1 ( a ti + b ti ) ] = E [ ∑ T + C u t =1 ( a ti + ∑ t -1 s =1 a si 1 [ s + c si ≥ t ] )] = E [ ∑ T t =1 a ti c ti ] = ¯ c i E [ ∑ T t =1 a ti ] for any i , where the last equality follows from the fact that c t is independent of a t . Hence, we have E [ ∑ T + C u t =1 ( a t + b t ) ] = D E [ ∑ T t =1 a t ] , where D ∈ R N × N > 0 denote the diagonal matrix with entries ¯ c 1 , ¯ c 2 , . . . , ¯ c N . As we have a t + b t ∈ A , this implies that 1 T + C u D E [ ∑ T t =1 a t ] = E [ 1 T + C u ∑ T + C u t =1 ( a t + b t ) ] is in the convex hull of A . We hence have ¯ r glyph[latticetop] E [ ∑ T t =1 a t ] = ( T + C u )( D -1 ¯ r ) glyph[latticetop] ( 1 T + C u D E [ ∑ T t =1 a t ]) ≤ ( T + C u ) max a ∈A { q glyph[latticetop] a } , [ ∑ ]

where the last inequality follows from D -1 ¯ r = q and the fact that 1 T + C u D E T t =1 a t is in the convex hull of A . Combining this with (13), we obtain the bound in Proposition 2.2.

A.2 Proof of Lemma 3.1

Proof. From (5), we have min i ∈ A ′ s T i ( t s ) ≥ C u /C l , which implies l s = C l min i ∈ A ′ s T i ( t s )+2 C u ≥ 2 C u . From this, for all i ∈ A ′ s . we have T i ( t s +1 ) -T i ( t s ) ≥ ( l s -2 C u ) /C u ≥ l s / (2 C u ) . This means the first inequality in Lemma 3.1 holds. Further, we have

which implies that the second inequality in Lemma 3.1 holds.

A.3 Proof of Lemma 3.2

Proof. Let i ∗ ∈ A ′ s be a task such that T i ∗ ( s ) = min i ∈ A ′ s T i ( t s ) . We then have l s = C l T i ∗ ( s )+2 C u . As the task i ∗ is completed at least ( l s -2 C u ) /C u = ( C l /C u ) T i ∗ ( s ) times within the s -th phase, we have T i ∗ ( s +1) -T i ∗ ( s ) ≥ ( C l /C u ) T i ∗ ( s ) , which implies T i ∗ ( s +1) ≥ (1 + C l /C u ) T i ∗ ( s ) . Further, if i ∈ A ′ s ′ for some s ′ < s , we have T ′ i ( s ) ≥ C u /C l as shown in the proof of Lemma 3.1. Hence, if s ≥ NK + 1 for some positive integer K , there exists i ∈ [ N ] such that T t i ( s ) ≥ ( C u /C l )(1 + C l /C u ) K -1 . From this, as t s ≥ C l T i ( t s ) holds for any i ∈ [ N ] , we have t s ≥ C l (1 + C l /C u ) s/N -2 .

A.4 Proof of Lemma 4.3

Let us first show that ∆ a ′ s ≤ ∑ i ∈ ˜ A s d i ( t s ) holds with a probability of at least 1 -6 N/t 2 s . Fix t arbitrarily.

For any i and let τ (1) < τ (2) < · · · represent all the indices of rounds at which the i -th task is started. Define the sequence E 1 , E 2 , . . . by E j = ∑ j s =1 ( r i ( τ ( s )) -¯ r i ) . Then, since E j is a sum of j i.i.d. random variables, the Azuma-Hoeffding inequality implies Pr[ | E j | ≥ √ 1 . 5 j · ln t ] ≤ 2 t for any t and j . We hence have

which means that the first part of (4) holds. Similarly, by considering E j = ∑ j s =1 ( c i ( τ ( s )) -¯ c i ) . it can be shown from Lemma A.1 that

We further have

where the first and second inequalities follow from the lower bound on T i ( t s ) in (5), and the third and last inequalities follow from C l ≤ ¯ c i ≤ C u . Hence, with probability 1 -6 N/t 2 ,

As (16) implies ¯ c i -2 d c i ( t ) ≥ ¯ c i / 2 , we have

where we denote ˜ r i = min { 1 , ¯ r i +2 d r i ( t ) } and ˜ c i = max { C l , ¯ c i -2 d c i ( t ) } ≥ ¯ c i / 2 . We here defined ˜ C i by

We then have

where the first and last inequalities follow from (17) and (18), respectively, and the second inequality follows from the definition of a ′ s . Indeed, a ′ s ∈ arg max a ∈A ˆ q ( t s ) glyph[latticetop] a implies the following inequality: ∑ i ∈ A ∗ \ A ′ s ˆ q i ( t s ) -∑ i ∈ A ′ s \ A ∗ ˆ q i ( t s ) = ˆ q ( t s ) glyph[latticetop] a ∗ -ˆ q ( t s ) glyph[latticetop] a ′ s ≤ 0 .

Let us next show the bound on R (1) T . We decompose R (1) T as follows:

where ¯ F s denotes the complement of the event F s . The second part of RHS can be bounded as

where the first inequality follows from the fact that (20) holds with a probability of at least 1 -6 N/t 2 s and the inequality of ∆ a ′ s ≤ M/C l . We have

where the last inequality follows from

Hence, we have

Combining (21), (22) and (23), we obtain the bound on R (1) T in Lemma 4.3.

A.5 Proof of Lemma A.2

The following lemma can be shown in a way similar to that of Kveton et al. [25, Lemma 3].

Lemma A.2. Let { α k } and { β k } be positive real sequences for which 1 = β 0 > β 1 > · · · > β k > · · · > 0 , α 1 > α 2 > · · · > α k > · · · > 0 , lim k →∞ α k = lim k →∞ β k = 0 , and ∑ ∞ k =1 β k -1 -β k √ α k ≤ 1 . Suppose that F s occurs. There then exists k such that S s,k = { i ∈ ˜ A s | ∆ a ′ s ≤ M √ α k d i ( t s ) } satisfies | S s,k | ≥ β k M .

Proof. Let us show the claim via proof by contradiction. Suppose that | S s,k | < β k M holds for all k . We then have

where the first inequality follows from the definition of S s,k , i.e., ˜ CM √ α k d i ( t s ) < ∆ a ′ s holds for all i ∈ [ N ] \ S s,k , and the second inequality follows from the initial assumption made in this proof. This means that F s occurs. Hence, if F s occurs, thi initial assumption is false, which means that there exists k such that | S s,k | ≥ β k M .

We will apply Lemma A.2 with { α k } and { β k } such that ∑ ∞ k =1 α k β k = O (1) , which are provided in Appendix A.4 in [25]. From Lemma A.2, if F s occurs, ∑ i ∈ ˜ A 1 [ i ∈ ˜ A s , ∆ a ′ s ≤ 2 ˜ CM √ α k d i ( t s )] ≥ β k M holds for some k , which implies

Proof. Using (24), we can bound ˆ R T as

Let us evaluate ∑ S s =1 l s 1 [ G s,k,i ] √ T i ( t s ) for some fixed k and i . If G s,k,i occurs, we have ∆ i ≤ ∆ a ′ s ≤ M √ α k d i ( t s ) ≤ M √ α k ˜ C i ln T ¯ c i T i ( t s ) which implies that √ T i ( t s ) ≤ M √ α k ˜ C i ln T ¯ c i 1 ∆ i =: γ . We hence have

The expectation of the right-hand side can be bounded as follows. From Lemma A.4 in the Appendix, we have

By using Lemma 3.1, we obtain T i ( t s +1 ) -T i ( t s ) √ T i ( t s ) = 3 T i ( t s +1 ) -T i ( t s ) √ 4 T i ( t s )+ √ T i ( t s ) ≤ 3 T i ( t s +1 ) -T i ( t s ) √ T i ( t s +1 )+ √ T i ( t s ) = 3( √ T i ( t s +1 ) -√ T i ( t s )) . Combining this with (26) and (27), we obtain E [ ∑ S s =1 l s 1 [ G s,k,i ] √ T i ( t s ) ] ≤ 6¯ c i ∑ S s =1 ( √ T i ( t s +1 ) -√ T i ( t s ) ) 1 [ √ T i ( t s ) ≤ γ ] ≤ 12¯ c i γ = 12 M √ α k ¯ c i ˜ C i ln T ∆ i , where we the last inequality follows from Lemma 3.1. Combining this with (25), we obtain E [ ˆ R T ] = O ( M ln T · ∑ i ∈ ˜ A ˜ C i ∑ ∞ k =1 α k β k 1 ∆ i ) = O ( M ∑ i ∈ ˜ A ˜ C i ln T ∆ i ) , which completes the proof.

A.6 Proof of the second part of (10) (bounding ˆ R (2) T )

The second part of (10) can be shown via the following:

Proof. For any s ≥ 1 and task i ∈ [ N ] , the number of times to start task i duaring the s -th phase is at least ( T i ( t s +1 ) -T i ( t s ) -1) . We hence have

where the second inequality follows from Lemma A.4 and the third inequality follows from q i = ¯ r i / ¯ c i ≤ 1 /C l and | A ′ s | ≤ M . This completes the proof.

We used the following lemma in the proof of Lemma A.3.

Lemma A.4. For any s ≥ 1 and i ∈ A ′ s , we have

where E s denote the conditional expectation given l s and T i ( t s ) .

Proof. We will show (28) by induction in the value of l s ≥ 0 . It is clear that (28) hold if l s ≤ 2 C u . As the inductive hypothesis, we assume that (28) holds if l s ≤ m for some m ≥ 2 C u . Then, if l s = m +1 > 2 C u , at least once, task i ∈ A ′ s is started and completed during the s -th phase. Let c i denote the processing time when task i is first started and completed during the s -th phase. We then have

where the inequality follows from the inductive hypothesis. Hence, (28) holds for l = m +1 as well. By induction, it has been shown that (28) holds. Further, as we have l s ≥ C l B +2 C u ≥ 4 C u , we have l s -2 C u ≥ l s / 2 , which implies (29) holds.

From Lemma A.3, we have ˆ R (2) T ≤ 2 C u C l SM . By combining this with S = O ( C u C l N ln T ) , which follows from Lemma 3.2, we obtain the second part of (10).

Consequently, we have

A.7 Proof of Theorem 4.5

We start from the special case in which M = 1 , i.e., we assume A = {{ i } | i ∈ [ N ] } ∪ {∅} . We will show the following two lower bounds separately:

These two bounds together lead to the regret lower bound of

In fact, if C u ≤ 2 C l , (30) implies that (32) holds as we have C u C 2 l = Θ( 1 C l ) . On the other hand, if C u ≥ 2 C l , (31) implies that (32) holds since we have C u -C l C l C u = Ω( 1 C l ) .

From the lower bound of (32), we can obtain

by the technique used in the proof of Proposition 2 by Kveton et al. [25], which provides a regret lower bound for combinatorial semi-bandits. To see this, suppose that N = MK for simplicity. The action set of K -path combinatorial semi-bandit (in Proposition 2 by Kveton et al. [25]) is expressed as A ′ = { A k | k ∈ [ K ] } ∪ {∅} , where A k = { M ( k -1) + j | j ∈ [ M ] } . Note that this does not satisfy the assumption that the action set is closed under inclusion, which is posed in our problem setting. In order to address this assumption, we set A = { A ⊆ [ N ] | ∃ A ′ ∈ A ′ , A ⊆ A ′ } , which is closed under inclusion. Let us consider bandit task assignment problems with action set A , instead of A ′ . We suppose that, for each k ∈ [ K ] , r ti and c ti take the same values for all i ∈ A i , i.e., there exists r ′ tk and c ′ tk such that r ti = r ′ tk and c ti = c ′ tk for all i ∈ A i . If r ′ _ tk and c ′ _ tk are i.i.d. for different t , this is a proper problem instance in our model. (Our model does not assume that r ti and c ti are independent for different tasks i as stated in Section 2). For any action A t ∈ A \ {∅} , let A ′ t ∈ A ′ be such that A t ⊆ A ′ t . Then, if ( A t ) T t =1 is feasible, ( A ′ t ) T t =1 is also feasible, and the reward for the latter is greater than or equal to the former. This means that any algorithm can be converted to one that takes only actions in A ′ without sacrificing the performance. Hence, it is sufficient to consider only algorithms that choose actions from A ′ in the proof of lower bounds. Therefore, regret lower bounds with action set {{ k } | k ∈ [ K ] } ∪ {∅} , multiplied with M , apply to our problem setting. Hence, the lower bound of (32) implies (33) as well.

The rest of this section is dedicated to the proofs of (30) and (31).

Proof of (30) If M = 1 and c ti = C l for all t and i with probability one, the problem is equivalent to a standard N -armed bandit problem with time horizon T ′ = Θ( T/C l ) . As shown in Theorem 5.1 by Auer et al. [6], any algorithm for the N -armed bandit problem suffers Ω(min { √ NT ′ , T ′ } ) -regret in the worst case with time horizon T ′ . By substituting T ′ = Θ( T/C l ) into this lower bound, we obtain (30).

Proof of (31) Set p = C l C u + C l . Fix an arbitrary algorithm. For some fixed i ∗ ∈ [ N ] and glyph[epsilon1] ∈ [0 , p/ 2] , define a distribution D i ∗ ,glyph[epsilon1] of c and r as follows:

We also denote this distribution with glyph[epsilon1] = 0 by D 0 , which does not depend on i ∗ . Denote ¯ c = pC u +(1 -p ) C l = 2 C l C u C l + C u = 2 pC u . Then the expected processing time can be expressed as

Denote the number of completing the task i during T rounds by T i . Then the expected regret can be expressed as

where E i ∗ ,glyph[epsilon1] means the expectation for the case in which c and r follow D i ∗ ,glyph[epsilon1] . As the expected value of the number of rounds to process the task i is expressed as E i ∗ ,glyph[epsilon1] [¯ c i T i ] , we have

From (34), the left-hand side of this can be expressed as

Combining the above, we obtain

Similarly, we can show that

where E 0 means the expectation for the case in which c and r follow D 0 . This means that there exists i ∗ such that

We assume that i ∗ satisfies this in the following.

From (35) and (36), we have

By a similar argument to Lemma 6.6 of by Badanidiyuru et al. [7], we can show that

Hence, if glyph[epsilon1] = min { p 4 , 1 16 √ p ¯ cN T }

Combining this with (37), we obtain

which completes the proof of (31).



√}√{̂

























√˜˜√˜√

}⌉̂⊎

∑√}√}√˜̂

B Notes on experimental setup

In the implementation on UCB-BV1 proposed by Ding et al. [15], by considering each element in A = { A ⊆ [ N ] | | A | = M } as a single arm, we transform an instance of the bandit task assignment into a budget constrained multi-armed bandit problem with ( N M ) -arms. To satisfy the constraints, we wait until all tasks in the previously chosen set are completed before deciding the next arm, i.e., the budget consumed by choosing A t is given as max i ∈ A t { c ti } .

Similarly, in the implementation of CombUCB1 proposed by Kveton et al. [25], each time a set A t in A is selected and it waits until all the tasks in A t are completed before selecting the next set. However, the way of choosing the set is different from that of UCB-BV1: we maintain the UCB scores for each task i ∈ [ N ] , not for each set A ∈ A , and then choose a set so that the sum of the UCB scores for tasks in it are maximized, as suggested in the paper by Kveton et al. [25]. Data and source code to reproduce the experimental results in this paper are included in the supplementary materials.

C Additional experiments

This section provides the results of numerical experiments performed to evaluate (i) the performance for randomly generated instances and (ii) how the algorithm behaves under different settings, such as varying values of N and C u . Settings of parameters not specified in the captions are the same as in Section 5 and in Appendix B.

Figures 4 and 5 show the results of synthetic experiments for randomly chosen ¯ r and ¯ c . This experiment confirms that the proposed algorithm works well empirically in randomly generated environments, including cases where the values of ¯ r are not necessarily constant.

Some results of an empirical evaluation of the algorithm's sensitivity to parameters N and C u are given in Figures 6, 7, and 8. Figure 6 suggests that the values of the regret scale linearly with respect to N , which is consistent with the regret upper bound of (7). Figure 7 shows the computation time for varying problem sizes, which implies that the proposed algorithm is superior in terms of computational complexity. Figure 8 represents how false beliefs about C u affect the performance of the algorithms. Experimental results suggest that the performance of the proposed algorithm is sensitive to underestimation of C u , while it is robust to overestimation of C u .



√}√{̂





















√˜˜√˜√

}⌉̂⊎

∑√}√}√˜̂

3

4

5

6

7

8

9

10

N: number of tasks

0

500

1000

1500

2000

2500

3000

3500

4000

regret

UCB-BV1

CombUCB1

Proposed

3

4

5

6

7

8

9

10

N: number of tasks

0.0

0.2

0.4

0.6

0.8

1.0

1.2

computational time [s]

UCB-BV1

CombUCB1

Proposed

2

4

6

8

10

Cu

: input parameter corresponding to

Cu

0

500

1000

1500

2000

2500

regret

UCB-BV1

CombUCB1

Proposed

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the *bandit task assignment* problem.\nIn this problem, a player chooses tasks sequentially,\neach task taking some random time to complete and giving some random reward.\nBoth random quantities are sampled from a priori unknown distributions.\nIn each round, the player may start a new task, as long as the set of active\ntasks satisfies a constraint, which can be modeled as an arbitrary subset $A \\subseteq 2^{[N]}$.\n\nThe problem is a generalization of the combinatorial semi-bandit\nand the budget constrained bandit problems.\nAssuming that each task takes only one time step gives the semi-bandit model while\nassuming that at each time only a single task is active gives the budget constrained bandit problem where\nthe *cost* of each task is its time and the *budget* is the total time T.\n\nThe authors propose an algorithm regret $\\tilde{O}(\\sqrt{MNT})$ where $T$ is the time horizon,\n$N$ is the number of tasks and $M$ is the maximum number of tasks that run at the same time.\nFirstly, the authors show how to obtain a nearly optimal policy using the expected per-round rewards for tasks.\nThey then use a UCB-based approach on the per-round rewards to design the algorithm.\n\nSTRENGTHS:\nThe paper is well-written studies an interesting problem which is a nice generalization of both combinatorial semi-bandits and the budget-constrained bandits.\nThe problem is well-motivated and the relation to related work is described very well.\n\nWEAKNESSES:\nThe techniques in the paper seem to be mostly based on prior work,\nthough I think this is a minor point.\n\nQUESTIONS:\nIn the beginning of section 3, the notation $b\\le a$ is initially confusing as the tasks are thought\nof as subsets rather than binary vectors. It would be better to add a short explanation to clarify this.\nFormalize footnote in page 4.\n\nRegarding the relation with blocking bandits,\nI think it is also good to compare with the special case of task assignment when there are no constraints as this also very similar to blocking bandits.\nIn this case, seems to me that the key distinction between the problems is that in blocking bandits only a single action can be played in each round, while in bandit task assignments and arbitrary number\nof tasks may be started at any moment.\nGiven that blocking bandits does not admit any pseudo-polynomial time algorithm, it may be good to mention why bandit task assignment can be *easier* than blocking bandits.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel bandit learning scenario, in which the learner can select in any time step tasks, each of which leads to stochastic rewards and processing time that blocks the task for several rounds. At any round, the tasks in progress (i.e., both newly started or still running ones) are constrained to be in a predefined given set of allowed tasks. The paper develops a solution to a regret minimization problem in this scenario and shows optimality of it by means of a lower bound.\n\nSTRENGTHS:\nThe paper introduces an interesting and well-motivated learning scenario, which it distinguishes in detail from existing related ones. The developed solution to the corresponding regret minimization scenario comes with a non-trivial regret bound with a detailed proof in the appendix. Moreover, a corresponding lower bound indicates its optimality.\nThe paper is well-written in terms of used notation and the presentation of the proofs, it has very little typos.\n\nWEAKNESSES:\n- In Thm. 5.1, the gap-free bound is of the form $O(X + \\frac{C_u^2}{C_l^2} NM\\ln T)$. However, in the later discussion (e.g. in line 326), this second summand seems to be ignored. Why? Note that its dependence on $M, N$ and $C_u$ is worse than that of term $X$. And in line 355 you explicitly mention this term.\n- As already mentioned by the authors, their solution requires knowledge of $C_u$ and $C_l$, which may not be possible in real-world. Can you discuss or test empirically the influence, which false beliefs of these values would have on the suffered regret?\n- The empirical evaluation of the proposed solution is very limited. The authors do not consider real-world data, and the synthetic experiments are limited to two scenarios with the same $\\bar{r}$. What about other values of $\\bar{r}$? Would it make sense to evaluate the algorithms on randomly chosen $\\bar{r}$, $\\bar{c}$?\n\nTypos:\n- 216: $\\min$ in the definition of $R_T$?\n- 262: $1-8/t^2$?\n- 352: wait\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors address the limitation of their work in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses thel problem of bandit task assignment, where a set of tasks need to be sequentially selected with uncertain processing times. The paper presents a UCB-type algorithm designed for bandit task assignment and provides theoretical analyses on upper bounds of the proposed algorithm and a regret lower bound of the problem. These bounds quantify the suboptimality of the algorithm and provide insights into its performance. The paper conducts numerical experiments to evaluate the proposed algorithm. It compares the proposed algorithm with existing approaches and demonstrates its superiority.\n\nSTRENGTHS:\n-\tThe paper investigates the bandit task assignment problem, which combines elements of task assignment and multi-armed bandit problems. \n-\tThe paper provides a rigorous theoretical analysis of the proposed algorithm, deriving regret bounds and a gap-free regret bound. The analysis is well-founded and based on solid mathematical reasoning, ensuring the reliability and validity of the presented results.\n-\tThe paper is well-written and well-organized. The concepts, algorithms, and theoretical analyses are presented in a clear and understandable manner. The paper also includes detailed explanations and supplementary material, enhancing the clarity and comprehensibility of the presented work.\n-\tBy addressing the bandit task assignment problem, the paper may be useful in tackling a practical challenge that arises in various real-world scenarios. The proposed algorithm and its regret bounds offer a promising solution for optimizing task assignment decisions in the presence of uncertain processing times. The numerical experiments provide empirical evidence of the algorithm's superiority compared to existing approaches, further emphasizing its significance and potential impact in practice.\n\nWEAKNESSES:\n- There exist related works that have not been addressed in the work, including:\nLearning to control renewal processes with bandit feedback, S Cayci, A Eryilmaz, R Srikant, 2019.\nBudget-constrained bandits over general cost and reward distributions, S Cayci, A Eryilmaz, R Srikant, 2020.\n-\tThe paper relies solely on synthetic datasets for the experimental evaluation, which may not fully capture the complexities and nuances of real-world scenarios. Incorporating real-world data or conducting experiments on real systems would provide a more comprehensive assessment of the algorithm's performance and validate its effectiveness in practical settings. Additionally, including case studies or application-specific examples would enhance the paper's relevance and demonstrate its potential impact in real-world contexts.\n-\tThe paper lacks a detailed analysis of the algorithm's computational complexity. Understanding the time and space complexity of the proposed algorithm is crucial for assessing its scalability and efficiency. Providing an in-depth analysis or empirical evaluation of the algorithm's computational requirements would help readers understand its feasibility for large-scale problems and enable comparison with other existing approaches.\n-\tThe paper does not provide a thorough sensitivity analysis of the algorithm's performance with respect to variations in key parameters. Understanding how the algorithm behaves under different settings, such as varying values of $C_u$, $C_1$, $N$, or $M$, would provide valuable insights into its robustness and generalizability. Conducting sensitivity analysis experiments and discussing the algorithm's performance across a range of parameter values would strengthen the paper's findings and help identify any limitations or constraints of the proposed approach.\n-\tThe paper focuses primarily on the theoretical aspects of the bandit task assignment problem and the proposed algorithm. However, it provides limited guidance on the practical implementation and deployment of the algorithm in real-world systems. Discussing practical considerations, such as handling dynamic environments, adapting to changing task characteristics, or integrating with existing task assignment frameworks, would provide valuable insights and enable readers to apply the algorithm in practical scenarios.\n\nQUESTIONS:\n-\tIn the conclusion (Section 7), the authors briefly mentioned the potential modification of the problem framework to address the assumption of known upper and lower bounds on processing times. It would be helpful to elaborate on possible modifications or extensions that could relax this assumption. Specifically, how would allowing for uncertainty or adaptability in the processing time bounds impact the algorithm design and analysis? Discussing potential modifications or extensions would provide insights into the algorithm's adaptability to dynamic or uncertain environments.\n-\tThe paper assumes that the bandit task assignment problem operates in a centralized setting where a single entity makes task assignment decisions. It would be helpful to consider or discuss potential extensions of the problem to a decentralized setting, where multiple agents collaborate to make task assignment decisions. Exploring the implications of decentralization and studying the design of distributed algorithms for task assignment could offer insights into real-world applications involving multiple decision-making entities.\n-\t(Minor comment) There is a missing period in line 267.\n\nLIMITATIONS:\n-\tThe authors address its potential limitations and future works in Section 7.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper introduces a new bandit problem where at each round the agent choses a subset of tasks to run, where the length of tasks and their rewards are unknow and revealed only after the task is done. A UCB style algorithm is proposed and shown to achieve almost-optmial regret bounds.

STRENGTHS:
1. The paper introduces a new bandit setting which generalizes quiet few other settings, so it has many potential applications in practice and theory.
2. The synthetic experiments demonstrate that the proposed algorithm immensely outperforms the related algorithms.
3. The analysis are well detailed and sketch proofs are included in the text.

WEAKNESSES:
1. The presentation could improve by moving the related works and side discussions to the later sections and instead focusing on the main setting introduction. It is hard to understand the setting since it is long laid in several pages.

QUESTIONS:
I suggest to summarize Section 1 and move Section 2 to after Section 5.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10