SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hanna Sumita
sumita@c.titech.ac.jp
Takuro Fukunaga
Chuo University
fukunaga.07s@g.chuo-u.ac.jp
Daisuke Hatano
RIKEN AIP
daisuke.hatano@riken.jp
Naonori Kakimura
kakimura@math.Keio.ac.jp
Ken-ichi Kawarabayashi
National Institute of Informatics, The University of Tokyo
k_keniti@nii.ac.jp
This study considers a novel problem setting, referred to as bandit task assignment, that incorporates the processing time of each task in the bandit setting. In this problem setting, a player sequentially chooses a set of tasks to start so that the set of processing tasks satisfies a given combinatorial constraint. The reward and processing time for each task follow unknown distributions, values of which are revealed only after the task has been completed. The problem generalizes the stochastic combinatorial semi-bandit problem and the budget-constrained bandit problem. For this problem setting, we propose an algorithm based on upper confidence bounds (UCB) combined with a phased-update approach. The proposed algorithm admits a gap-dependent regret upper bound of  $O(MN(1 / \Delta)\log T)$  and a gap-free regret upper bound of  $\tilde{O} (\sqrt{MNT})$ , where  $N$  is the number of the tasks,  $M$  is the maximum number of tasks run at the same time,  $T$  is the time horizon, and  $\Delta$  is the gap between expected per-round rewards of the optimal and best suboptimal sets of tasks. These regret bounds nearly match lower bounds.
This paper introduces a new model of sequential decision-making that we refer to as the bandit task assignment problem. The goal of this model is to determine which tasks to perform sequentially so that the total reward will be maximized, while estimating the distribution of rewards and processing time for each task. For each round  $t = 1,2,\dots ,T$ , we choose a set  $A_{t}$  of tasks to start. Each task  $i$  in  $A_{t}$  will be completed in the  $(t + c_{ti})$ -th round, and we then obtain the reward  $r_{ti}$ . The processing time  $c_{ti}$  and the reward  $r_{ti}$  follow unknown distributions independently for all  $t$ , and they are observed only when the task has been completed, i.e., only bandit feedback is available. In all rounds, the set of processing tasks is required to satisfy a given combinatorial constraint. That is, in each round, the player can start a set of new tasks that, together with the tasks still in progress, satisfies the given
Figure 1: An example of a feasible task assignment.
constraint. The performance of the algorithm is evaluated by means of regret  $R_{T}$ , which is defined as the difference between the cumulative rewards obtained with an optimal policy and with the algorithm. Our model allows an arbitrary constraint for a set of tasks that is expressed by  $\mathcal{A} \subseteq \{0,1\}^N$ , where  $N$  is the number of tasks. Each vector  $a = (a_{1},a_{2},\ldots ,a_{N})^{\top} \in \mathcal{A}$  is interpreted as a feasible set of tasks  $A = \{i\in [N]\mid a_i = 1\}$ , where  $[N] \coloneqq \{1,2,\dots,N\}$ .
Figure 1 illustrates an example of a feasible task assignment for the case of  $\mathcal{A} = \{a\in \{0,1\} ^3\mid$ $\| a\| _1\leq 2\}$ , in which we choose which tasks to start from three candidates so that the number of processing tasks does not exceed two. Each arrow in this figure represents the interval of time to process a task. In this example, at round 2, two tasks (Tasks 1 and 2) are still processing, and hence, we cannot start any new task. At the beginning of round 3, we observe that Task 1 is completed, and it is then possible to start one new task. We note that the processing time  $c_{ti}$  and the reword  $r_{ti}$  of task  $i$  stated at time  $t$  are revealed only at the completion time  $(t + c_{ti})$ .
The model in this paper is a common generalization of stochastic combinatorial semi-bandit problems [25, 24, 34] and budget-constrained bandit problems [20, 15, 12, 37]. In fact, if the processing time  $c_{ti}$  is always equal to 1 for any task  $i$ , the player can choose any set of  $\mathcal{A}$  in every round and can observe the feedback immediately afterward, which coincides with the stochastic combinatorial semi-bandit problem. On the other hand, if we suppose that  $\mathcal{A} = \{a \in \{0,1\}^N \mid \| a \|_1 \leq 1\}$ , i.e., if only a single task can be performed in every round, the problem corresponds to stochastic  $N$ -armed bandits with a budget constraint [20, 15]. Indeed, by regarding  $c_{ti}$  as the cost for choosing arm  $i$ , we can interpret the model as a problem with the budget constraint that the cumulative cost is at most  $T$ .
We note that our problem setting is similar to blocking bandits [8], in which each chosen arm is blocked, i.e., is unavailable for a certain period of time after it is chosen. However, they are different in the constraints on the actions in each round. For example, consider  $\mathcal{A} = \{a\in \{0,1\} ^N\mid \| a\| _1\leq K\}$ . Then, in the bandit task assignment problem, we cannot start a new task when  $K$  tasks are already running. On the other hand, in the blocking bandit problem, the player can choose any  $K$  arms that are not blocked. Blocking bandits have been extended to contextual bandit models [9], semi-bandit models [3, 28], and adversarial models [10], as well.
Applications of our model include the following examples:
Example 1 (Matching constraint). We consider the problem of assigning  $L$  jobs to  $K$  workers. Let  $V_{1}$  and  $V_{2}$  be the sets of workers and jobs, respectively. Let  $E \subseteq V_{1} \times V_{2}$  be a set of pairs of a worker and a job such that  $(u,v) \in E$  if and only if the worker  $u$  is qualified to do the job  $v$ . In each round, we choose a subset  $M$  of  $E$  so that no two elements in  $M$  share the same worker or job, that is,  $M$  must satisfy the matching constraint for the bipartite graph  $G = (V_{1} \cup V_{2}, E)$ . Each job has processing time to complete, and, when a job is assigned to a worker, the job-worker pair is occupied until the job is completed. Thus the occupied job-worker pairs have to be included in  $M$ . We assume that the time required to complete a job and the quality of the results (rewards) follow unknown distributions that depend on the job-worker pair. This problem can be regarded as a special case of the bandit task assignment problem with  $N = |E|$ . Choosing  $a_{t} \in \mathcal{A}$  corresponds to assigning a job  $v$  to a worker  $u$  for each  $e = (u,v)$  such that  $a_{te} = 1$ , and the worker  $u$  and the job  $v$  will then be occupied and unavailable until the task is completed. It would be worth mentioning that the matching constraint is often the subject of consideration in the context of combinatorial semi-bandits [18, 30].
Example 2 (Knapsack constraint). Consider the situation where we want to perform tasks (e.g., of data processing) under resource constraints (e.g., computational resources such as CPUs, GPUs, and memories). We assume that the processing time of each task varies stochastically, and is not revealed until it is completed. Formally, we are given  $N$  tasks and  $K$  resources. Assume that each task  $i \in [N]$  consumes  $b_{ki} \in \mathbb{R}_{\geq 0}$  amount of resource  $k \in [K]$  per unit time, and that the total
amount of resource  $k$  available in unit time is  $f_{k}$ . Then the player can choose a set of tasks from  $\mathcal{A} = \{a\in \{0,1\} ^N\mid b_k^\top a\leq f_k\quad (k\in [K])\}$ , where  $b_{k} = (b_{k1},b_{k2},\ldots ,b_{kN})^{\top}\in \mathbb{R}_{\geq 0}^{N}$  for each resource  $k$ . Note that the set of all running tasks must satisfy the constraint given by  $\mathcal{A}$ , which means that we cannot use resources occupied by tasks in progress and we are restricted to choose a set of tasks that can be executed with remaining resources. We note that the class of problems with a single resource  $(K = 1)$  includes the budget-constrained multi-armed bandit [20, 15] as a special case.
Example 3 (Matroid constraint [35, 17]). If  $\mathcal{A}$  consists of indicator vectors for independent sets of a matroid with the ground set  $[N]$ , we call this a matroid constraint. Matroid constraints capture a variety of problem settings of practical importance, including the situation in which the number of tasks that can be executed simultaneously is limited to at most  $K$ , i.e.,  $\mathcal{A} = \{a \in \{0,1\}^N \mid \|a\|_1 \leq K\}$ . This special case is called a uniform matroid.
1.1 Contribution
Our main contribution is to present an algorithm that achieves a regret bound of  $O\left(\frac{C_{\mathrm{u}}}{C_{1}^{2}} MN(1 / \Delta + C_{\mathrm{u}}) \ln T\right)$ , where  $C_{\mathrm{u}}$  and  $C_{1}$  are upper and lower bounds for the processing times  $c_{ti}$ ,  $M$  is the maximum of  $\|a\|_{1}$  for  $a \in \mathcal{A}$ , and  $\Delta > 0$  is the gap between expected per-round rewards for the optimal and the best suboptimal actions. The proposed algorithm also enjoys a gap-free regret bound of  $O\left(\frac{1}{C_{1}} \sqrt{C_{\mathrm{u}} M N T \ln T}\right)$ . Our contributions include a nearly tight lower bound as well, i.e., we show that any algorithm suffers a regret at least  $\Omega\left(\frac{1}{C_{1}} \sqrt{C_{\mathrm{u}} M N T}\right)$ . This indicates that the proposed algorithm achieves nearly optimal regret bounds.
To design our algorithm, we first show that a nearly optimal policy can be characterized by expected per-round rewards for tasks (Proposition 2.2 in Section 2). This implies that, given estimation for expected per-round rewards and a linear optimization oracle for  $\mathcal{A}$ , we can compute asymptotically optimal actions. This is in contrast to the blocking bandits, in which it is computationally hard to maximize the total reward even if all the distributions are known, as presented, e.g., in Corollary 3.2 by Basu et al. [8]. That is why, for the blocking bandits, existing algorithms can achieve only approximate regret bounds, i.e., the performance is only guaranteed to be (close to) a constant-factor approximation for the optimal strategy.
Proposition 2.2 suggests us to balance exploration and exploitation with the aid of upper confidence bounds (UCB) on the expected per-round reward for each task, which can be computed efficiently. Such a UCB-based approach is similar to CombUCB1 for combinatorial semi-bandits [25]. CombUCB1, however, does not directly apply to our problem setting due to the stochastically variable processing time. In the bandit task assignment problem, the set of tasks that the player can start in each round changes depending on the set of tasks still in progress, which makes the decision of the player restricted. To address this issue, we introduce a phased-update approach, in which we divide the entirety of the rounds  $[T] = \{1,2,\dots,T\}$  into segments of appropriate length. At the beginning of each segment  $s$ , we compute a set of tasks  $A_{s}^{\prime}$  based on the UCB for expected per-round rewards. We note that this can be done independently of the state of running tasks. Then, in the  $s$ -th segment, we basically continue to perform the same set of tasks  $A_{s}^{\prime}$ .
One main technical difficulty is how to determine the length  $l_{s}$  of the segment  $s$ . At the start of a new segment  $s$ , tasks in  $A_{s}^{\prime}$  cannot be executed until all tasks performed in the previous segment have been completed, i.e., every switching point between segments causes idle time with non-zero probability. Therefore, the larger the number of segments (i.e., the shorter the length of each segment), the greater the loss due to such waiting time. In particular, if the segment length is  $O(C)$ , we cannot avoid a linear regret of  $\Omega(T / C)$  due to idle time. On the other hand, setting too long segments also has a negative impact on performance as the longer the length of each segment, the less frequently the set  $A_{s}^{\prime}$  will be updated, which may degrade the efficiency of exploration and exploitation. To address this dilemma, this study designs a way of determining the segment length  $l_{s}$  by which these potential losses will be well balanced and the overall regret will be bounded as desired.
Another nontrivial technique in the proposed algorithm is to employ Bernstein-type confidence bounds [4, 27] for the processing time. Thanks to this, we can construct a tight UCB estimator for the expected per-round reward for each task  $i$ , with a width depending on the mean  $\bar{c}_i$  and variance  $\sigma_i^2$  of the processing time. This is essential to achieve the nearly optimal  $\tilde{O}(\frac{1}{C_1}\sqrt{C_{\mathrm{u}}MNT})$ -regret bound. In fact, as we mention in Remark 4.2, an algorithm with standard confidence bounds will lead to regret upper bounds with additional  $C_{\mathrm{u}} / C_1$  factors, which do not match the lower bound.
Summary of contributions The contributions of this paper are summarized in the following four points: (i) We propose a novel problem setting of the bandit task assignment, which incorporates simultaneously combinatorial constraints among tasks, (unknown) processing time of each task, and the exploration-exploitation trade-off. The model includes various practical applications as described in Examples 1, 2 and 3. (ii) We show that a nearly optimal policy can be characterized with expected per-round rewards, which can be computed efficiently given only a linear optimization oracle. This contrasts with the computational difficulty in blocking bandits. (iii) We provide an algorithm with nearly optimal regret bounds. We handle the difficulties arising from the combinatorial constraints together with processing times by the phased-update approach. (iv) We present a regret lower bound that matches to the regret bound of the proposed algorithm, ignoring logarithmic factors. This means that the proposed algorithm achieves nearly optimal performance in terms of the worst-case analysis.
Let  $N$  be the number of tasks. The player is given a family of feasible sets of tasks, which is expressed by  $\mathcal{A} \subseteq \{0,1\}^N$ . Here, each binary vector  $a$  in  $\{0,1\}^N$  corresponds to a set of tasks, and  $a \in \mathcal{A}$  means that the set of tasks  $A = \{i \in [N] \mid a_i = 1\}$  can be executed simultaneously. We assume that  $\mathcal{A}$  is closed under inclusion, i.e.,  $a \in \mathcal{A}$  and  $b \leq a$  together imply  $b \in \mathcal{A}$ . Note here that, for any vectors  $a = (a_1, \ldots, a_N)^\top$ ,  $b = (b_1, \ldots, b_N)^\top \in \mathbb{R}^N$ , the notation of  $b \leq a$  means that  $b_i \leq a_i$  holds for all  $i \in \{1, \ldots, N\}$ . We denote  $M = \max_{a \in \mathcal{A}} \|a\|_1$ . In each round  $t$ , the player will choose a set of tasks  $a_t$  from  $\mathcal{A}$ . The chosen set has to exclude all the processing tasks not yet completed. More precisely,  $a_t$  is constrained to satisfy  $a_t + b_t \in \mathcal{A}$ , where  $b_t$  is the set of tasks still in progress at the  $t$ -th round (see (1) below). Each task  $i$  with  $a_{ti} = 1$  is then started and will be completed at the beginning of the  $(t + c_{ti})$ -th round, which yields the reward of  $r_{ti}$ . The reward  $r_{ti}$  and the processing time  $c_{ti}$  will be revealed only after the task is completed. We assume that  $r_{ti}$  and  $c_{ti}$  are restricted so that  $r_{ti} \in [0,1]$  and  $c_{ti} \in \{C_1, C_1 + 1, \ldots, C_u\}$ , where  $C_1$  and  $C_u$  are integers satisfying  $1 \leq C_1 \leq C_u$ . We also assume that  $((r_{ti}, c_{ti}))_{i=1}^N$  follows an unknown distribution  $\mathcal{D}$  over  $([0,1] \times \{C_1, \ldots, C_u\})^N$ , independently for  $t = 1, 2, \ldots, T$ . Note that  $r_{ti}$  and  $c_{ti}$  may be dependent. From the above problem definition, a family  $\mathcal{A}_t$  of task sets available at the  $t$ -th round is expressed as follows:
$$
b _ {t i} = \sum_ {s = 1} ^ {t - 1} a _ {s i} \mathbf {1} [ s + c _ {s i} > t ] \quad (i \in [ N ]), \quad \mathcal {A} _ {t} = \{a \in \mathcal {A} \mid a + b _ {t} \in \mathcal {A} \}, \tag {1}
$$
where the vector  $b_{t} = (b_{ti})_{i=1}^{N}$  corresponds to the set of uncompleted tasks that starts before the  $t$ -th round. The goal of the player is to maximize the sum of rewards earned by the  $T$ -th round, which can be expressed as  $\sum_{t=1}^{T} r_{t}^{\top} a_{t}$ .
Remark 2.1. Stochastic combinatorial semi-bandits [25] and the multi-armed bandit problem with a budget constraint [15] are special cases of the bandit task assignment problem. In fact, if  $c_{ti} = 1$  for all  $t$  and  $i$ , the problem corresponds to a combinatorial semi-bandit problem with action set  $\mathcal{A}$ . In addition, if  $\mathcal{A} = \{a \in \{0,1\}^N \mid \|a\|_1 \leq 1\}$ , the problem corresponds to an  $N$ -armed bandit with budget  $B = T$  and costs  $\{c_{ti}\}$ .
In the problem, we assume that we are able to solve linear optimization over  $\mathcal{A}$  efficiently. More precisely, we suppose that we are given access to an offline linear optimization oracle that returns a solution  $\hat{a} \in \arg \max_{a \in \mathcal{A}} q^\top a$  for any input  $q \in \mathbb{R}_{>0}^N$ . Note here that, for the problems with matching constraints or with matroid constraints (Examples 1 and 3, respectively), such an oracle can be implemented in polynomial time [23, 17]. For Example 2, linear optimization over  $\mathcal{A}$  is NP-hard, but can be solved practically fast in many cases, with the aid of dynamic programming or integer programming algorithms.
We define the regret  $R_{T}$  to be the expectation of the difference between the total rewards obtained with the optimal proper policy and those obtained with the algorithm by the  $T$ -th round, where we call a policy for choosing  $a_{t}$  proper if  $a_{t}$  satisfies the constraint defined by (1) and each  $a_{t}$  depends on information observed by the beginning of  $t$ -th round, but not on  $((r_s, c_s))_{s=t}^T$ . Any proper policy can be expressed as sequences of mapping  $\pi = (\pi_t)_{t=1,2,\ldots}$ , where each  $\pi_t$  maps from the history  $h_{t}$  which consists of chosen actions  $(a_s)_{s=1}^{t-1}$  and feedback obtained by the beginning of the  $t$ -round, to
action  $a_{t}$  in  $\mathcal{A}$ . Let  $\Pi$  denote the set of all proper policies. The regret can then be expressed as
$$
R _ {T} = \max  _ {\pi^ {*} \in \Pi} \mathbf {E} \left[ \sum_ {t = 1} ^ {T} r _ {t} ^ {\top} \pi_ {t} ^ {*} \left(h _ {t} ^ {*}\right) \right] - \mathbf {E} \left[ \sum_ {t = 1} ^ {T} r _ {t} ^ {\top} a _ {t} \right],
$$
where  $(h_t^*)_{t = 1,2,\ldots}$  denotes histories generated by the policy  $\pi^{*}$ . Intuitively, the total reward obtained with the optimal proper policy corresponds to the maximum performance achieved by an agent with unlimited computational resources who knows the distribution  $\mathcal{D}$  of processing times and rewards. Note that the regret  $R_{T}$  is defined on the basis of tasks started by the  $T$ -th round. Although it may seem more natural to define the regret by the rewards for tasks completed by the  $T$ -th round, the difference between these two alternatives is only  $O(M / C_1)$ , which is almost negligible as it is independent of  $T$ .
We can see that the performance of the optimal proper policy (and therefore also regret) can be characterized by using the expected per-round rewards. We define  $\bar{r}_i$  and  $\bar{c}_i$  to be the expectation of  $r_{ti}$  and  $c_{ti}$ , respectively. We also denote  $q_i = \bar{r}_i / \bar{c}_i$ . Define vectors  $\bar{r} \in [0,1]^N$ ,  $\bar{c} \in [C_1,C_\mathrm{u}]^N$  and  $q \in [0,1/C_1]^N$  by  $\bar{r} = (\bar{r}_1,\bar{r}_2,\dots,\bar{r}_N)^\top$ ,  $\bar{c} = (\bar{c}_1,\bar{c}_2,\dots,\bar{c}_N)^\top$  and  $q = (q_1,q_2,\dots,q_N)^\top = (\bar{r}_1/\bar{c}_1,\bar{r}_2/\bar{c}_2,\dots,\bar{r}_N/\bar{c}_N)^\top$ . Note that  $q_i$  corresponds to the expected per-round reward for running each task  $i$ . Then, the expected total reward for an optimal policy can be bounded as follows:
Proposition 2.2. For any proper policy  $\pi = (\pi_t)_{t=1,2,\ldots} \in \Pi$ , we have  $\mathbf{E}\left[\sum_{t=1}^{T} r_t^\top \pi_t(h_t)\right] \leq (T + C_{\mathrm{u}}) \max_{a \in \mathcal{A}} \left\{q^\top a\right\}$ , where  $(h_t)_{t=1}^T$  denote histories generated by  $\pi$ .
Proposition 2.2 can be considered as a generalization of Lemma 1 by Ding et al. [15]. From this proposition, the regret can be bounded as follows:
$$
R _ {T} \leq \left(T + C _ {\mathrm {u}}\right) \max  _ {a \in \mathcal {A}} q ^ {\top} a - \mathbf {E} \left[ \sum_ {t = 1} ^ {T} r _ {t} ^ {\top} a _ {t} \right] \leq \mathbf {E} \left[ \sum_ {t = 1} ^ {T} \left(q ^ {\top} a ^ {*} - r _ {t} ^ {\top} a _ {t}\right) \right] + \frac {C _ {\mathrm {u}} M}{C _ {\mathrm {l}}}, \tag {2}
$$
where we denote  $a^* \in \arg \max_{a \in A} q^\top a$  and the last inequality follows from the facts that  $q_i \leq 1 / C_1$  holds for any  $i$  and that  $\|a^*\|_1 \leq M$ .
The proposed algorithm employs a phased-update approach, in which we update a policy for each phase, and each phase continues to select the same set of tasks.
The procedure starts with the initialization phase (or the zeroth phase), in which we execute each task  $B = \Theta \left( \frac{C_{\mathrm{u}}}{C_{1}} \ln T \right)$  times. This is required to ensure that the UCB estimators  $\hat{q}_i(t)$  given in (3) are close enough to the true expected reward  $q_i$ . Let  $t_1 \geq 1$  denote the round this initialization terminates, which is at most  $O(C_{\mathrm{u}}NB)$ . As the regret per round is at most  $\frac{M}{C_1}$ , the total regret in the initialization phase is at most  $O\left(\frac{C_{\mathrm{u}}MN B}{C_1}\right) = O\left(\frac{C_{\mathrm{u}}^2}{C_1^2} MN \ln T\right)$ , which turns out to be negligible in the overall regret upper bound.
The  $s$ -th phase  $(s \in \{1, 2, \ldots\})$  consists of a segment of rounds with length  $l_s$ , which implies that the first round  $t_s$  of the  $s$ -th phase is expressed as  $t_s = \sum_{u=1}^{s-1} l_u + t_1$ . At the beginning of each phase  $s$ , we compute  $a_s' \in \mathcal{A}$  and the length  $l_s$  of the  $s$ -th phase, for which the details will be described later. Then, in the subsequent  $l_s$  rounds, we continue to choose the set  $A_s' = \{i \in [N] \mid a_{si}' = 1\}$ . More precisely, in the  $t$ -th round, we choose  $a_t = a_s' - b_t$  if  $b_t \leq a_s'$  and choose  $a_t = 0$  otherwise, where we recall  $b_t$  is the set of tasks in progress at the  $t$ -th round (see (1)). Since  $a_s' \in \mathcal{A}$ ,  $a_t$  clearly belongs to  $\mathcal{A}_t$ . In this procedure, intuitively, we repeatedly restart each task  $i$  in  $A_s'$  just after  $i$  is completed, in the  $s$ -th phase. Note here that all tasks executed in the  $(s-1)$ -st phase are completed by the  $(t_s + C_u)$ -th round as the processing times are at most  $C_u$ , which implies that  $b_t \leq a_s'$  holds for any  $t \in [t_s + C_u, t_{s+1} - 1]$ . Hence, in any round  $t \in [t_s + C_u, t_{s+1} - 1]$ , it holds that  $b_t + a_t = a_s'$ , which implies all tasks in  $A_s'$  are running in the phase. Furthermore, for each task  $i \in A_s'$ , the number of times the task  $i$  is completed in the  $s$ -th phase will be at least  $\lfloor (l_s - C_u) / C_u \rfloor (\geq l_s / C_u - 2)$  and at most  $l_s / C_1 + 1$ .
We next describe how to compute  $a_{s}^{\prime}$ . We use upper confidence bounds (UCB) for  $q_{i} = \bar{r}_{i} / \bar{c}_{i}$ , the expected reward per round. In our definition of the UCB, we use the empirical means  $\hat{r}_i(t)$  and  $\hat{c}_i(t)$
of rewards and costs, the empirical variance  $V_{i}^{c}(t)$  of costs, and the number  $T_{i}(t)$  of times that the task  $i$  has been completed before the  $t$ -th round. We define the UCB  $\hat{q}_i(t)$  on  $q_{i}$  by
$$
\begin{array}{l} \hat {q} _ {i} (t) = \frac {\min \{1 , \hat {r} _ {i} (t) + d _ {i} ^ {r} (t) \}}{\max \{C _ {1} , \hat {c} _ {i} (t) - d _ {i} ^ {c} (t) \}}, \mathrm {w h e r e} d _ {i} ^ {r} (t) = \sqrt {\frac {1 . 5 \ln t}{T _ {i} (t)}}, \\ d _ {i} ^ {c} (t) = \sqrt {\frac {3 V _ {i} ^ {c} (t) \ln t}{T _ {i} (t)}} + \frac {9 \left(C _ {\mathrm {u}} - C _ {1}\right) \ln t}{T _ {i} (t)}. \tag {3} \\ \end{array}
$$