SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We would like to note this definition includes an empirical Bernstein bound  $d_i^c (t)$  [4] for the processing time  $c_{ti}$ , which will turn out to be essential for tight regret upper bounds, as mentioned in Remark 4.2. We then have
$$
\Pr \left[ \left| \hat {r} _ {i} (t) - \bar {r} _ {i} \right| \geq d _ {i} ^ {r} (t) \right] \leq \frac {2}{t ^ {2}}, \quad \Pr \left[ \left| \hat {c} _ {i} (t) - \bar {c} _ {i} \right| \geq d _ {i} ^ {c} (t) \right] \leq \frac {4}{t ^ {2}}. \tag {4}
$$
These can be shown from Azuma-Hoeffding inequality and empirical Bernstein's inequality [4], of which details can be found in Section A.4 of the appendix. Hence, with probability  $1 - 6 / t^2$ , we have  $q_{i} \leq \hat{q}_{i}(t)$ . At the beginning of each phase  $s$ , i.e., in the  $t_s$ -th round, we calculate these UCBs  $\hat{q}(t_s) = (\hat{q}_1(t_s), \hat{q}_2(t_s), \dots, \hat{q}_N(t_s))^{\top}$ , and then call the linear optimization oracle to find  $a_s' \in \arg \max_{a \in \mathcal{A}} \hat{q}(t_s)^{\top} a$ .
We set the length  $l_{s}$  of the  $s$ -th phase on the basis of  $\min_{i\in A_s'}T_i(t_s)$ , where we denote  $A_{s}^{\prime} = \{i\in [N]\mid a_{si}^{\prime} = 1\}$ . We set  $l_{s} = C_{1}\min_{i\in A_{s}^{\prime}}T_{i}(t_{s}) + 2C_{\mathrm{u}}$ . In the analysis, we assume that  $B$  is given by  $B = \lceil 90\frac{C_{\mathrm{u}}}{C_{1}}\ln T\rceil$ . We then have
$$
T _ {i} \left(t _ {s}\right) \geq 9 0 C _ {\mathrm {u}} \ln T / C _ {1}, \quad l _ {s} \geq 9 0 C _ {\mathrm {u}} \ln T, \quad 9 0 C _ {\mathrm {u}} N \ln T \leq t _ {1} = O \left(C _ {\mathrm {u}} ^ {2} N \ln T / C _ {1}\right), \tag {5}
$$
which will be used at several points in our analysis. These conditions and the construction of  $l_{s}$  lead to the bound on  $T_{i}(t_{s})$  as in the lemma below, which will be used in the regret analysis for the proposed algorithm.
Lemma 3.1. For any  $s \geq 1$  and  $i \in A_s'$ , we have  $l_s \leq 2C_{\mathrm{u}}(T_i(t_{s+1}) - T_i(t_s))$  and  $T_i(t_{s+1}) \leq 4T_i(t_s)$ .
All omitted proofs are given in the appendix. We also have the following bound on  $t_s$ :
Lemma 3.2. For any  $s \geq 1$ , there exists  $i \in A_s'$  such that  $T_i(t_{s+1}) \geq (1 + C_1 / C_u) T_i(t_s)$ . Consequently, we have  $t_s \geq C_1 (1 + C_1 / C_u)^{s/N-2}$  for any  $s \geq 1$ .
From Lemma 3.2, we have  $\ln t_s \geq \ln C_1 + \left(\frac{s}{N} - 2\right) \ln \left(1 + \frac{C_1}{C_u}\right) \geq \ln C_1 + \left(\frac{s}{N} - 2\right) \frac{C_1}{2C_u}$ , where the last inequality follows from the fact that  $\ln (1 + x) \geq x / 2$  for any  $x \in [0,1]$ . This implies that we have  $s \leq N\left(\frac{2C_u}{C_1} \ln t_s + 2\right)$ . Hence, the number of phases, denoted by  $S$ , is bounded by  $S \leq N\left(\frac{2C_u}{C_1} \ln T + 2\right) + 1 = O\left(\frac{C_u}{C_1} N \ln T\right)$ , since the last phase contains  $T$ .
The overall procedure of the proposed algorithm is summarized in Algorithm 1, which consists of arithmetic operations and offline linear optimization over  $\mathcal{A}$ . The number of arithmetic operations in each round is bounded by  $O(N)$  since the update of each parameter in Step 7-10 of Algorithm 1 can be performed in  $O(1)$ -arithmetic operations. Furthermore the number of calls to the offline linear optimization oracle is at most  $S = O\left(\frac{C_{\mathrm{u}}}{C_1} N \ln T\right)$ . In fact, the offline linear optimization oracle is called only at the beginning of each phase. This means that Algorithm 1 is more efficient than standard UCB algorithms for combinatorial semi-bandits algorithms [24, 25] that require  $O(T)$  calls to the oracle. The space complexity, other than that required for the offline optimization oracle, is  $O(N)$  since the algorithm works by maintaining  $O(N)$  parameters.
4 Regret Analysis
We denote  $a^* \in \arg \max_{a \in \mathcal{A}} q^\top a$ . Let  $A^* = \{i \in [N] \mid a_i^* = 1\}$  and  $\tilde{A} = [N] \setminus A^*$ . We define suboptimality gaps  $\Delta_a$ ,  $\Delta_i$  and  $\Delta_{\min}$ , respectively, by
$$
\Delta_ {a} = q ^ {\top} a ^ {*} - q ^ {\top} a \quad (a \in \mathcal {A}), \quad \Delta_ {i} = \min  _ {a \in \mathcal {A}: a _ {i} = 1, \Delta_ {a} > 0} \Delta_ {a} \quad (i \in [ N ]), \quad \Delta_ {\min } = \min  _ {i \in \tilde {A}} \Delta_ {i}. \tag {6}
$$
Algorithm 1 Phased-update UCB algorithm for bandit task assignment
Require:  $\mathcal{A} \subseteq \{0,1\}^N$ : a family of feasible task sets.  $C_1, C_u$ : lower and upper bounds for the processing time.  $B$ : length of initialization phase. Offline linear optimization oracle for  $\mathcal{A}$ .
1: For each  $i \in [N]$ , execute the  $i$ 's task  $B$  times, and let  $t_1$  be the round it completed. Set  $T_i(t_1) = B$ .
2: for  $s = 1,2,\ldots ,\mathbf{d}\mathbf{o}$
3: Set  $\hat{q}_i(t_s)$  by (3).
4: Call the offline linear optimization oracle to find  $a_{s}^{\prime} \in \arg \max_{a \in \mathcal{A}} \hat{q}(t_{s})^{\top} a$ , and set  $A_{s}^{\prime} = \{i \in [N] \mid a_{si}^{\prime} = 1\}$ .
5: Set  $l_{s} = C_{1}\min_{i\in A_{s}^{\prime}}T_{i}(t_{s}) + 2C_{\mathrm{u}}$  and  $t_{s + 1} = t_s + l_s$ .
6: for  $t = t_s, t_s + 1, \ldots, t_{s+1} - 1$  do
7: If  $b_{t} \leq a_{s}^{\prime}$ , output  $a_{t} = a_{s}^{\prime} - b_{t}$ , where  $b_{t}$  is given by (1). Otherwise, output  $a_{t} = 0$ .
8: for  $i = 1,2,\dots ,N$  do
9: If task  $i$  is completed (i.e.,  $b_{ti} = 0$ ) and  $c_{t',i}$  and  $r_{t',i}$  are observed ( $t'$  is the last round the task  $i$  is started), update  $\bar{c}_i(t)$ ,  $\bar{r}_i(t)$  and  $V_i^c(t)$  by using  $c_{t',i}$  and  $r_{t',i}$ . Set  $T_i(t + 1) = T_i(t) + 1$ .
0: end for
1: end for
2: end for
We also denote the variance of the processing time  $c_{ti}$  by  $\sigma_i^2 = \mathbf{E}[(c_{ti} - \bar{c}_i)^2]$ . We note that  $\sigma_i^2 \leq \mathbf{E}[(c_{ti} - C_1)^2] \leq (C_\mathrm{u} - C_1)\mathbf{E}[c_{ti} - C_1] = (C_\mathrm{u} - C_1)(\bar{c}_i - C_1)$ .
The goal of this section is to show the following:
Theorem 4.1. The regret for Algorithm 1 is bounded as
$$
R _ {T} \leq O \left(M \sum_ {i \in \tilde {A}} \frac {\tilde {C} _ {i} \ln T}{\Delta_ {i}} + \frac {C _ {\mathrm {u}} ^ {2}}{C _ {\mathrm {l}} ^ {2}} N M \ln T\right) \leq O \left(\left(\frac {1}{\Delta_ {\min }} + C _ {\mathrm {u}}\right) \frac {C _ {\mathrm {u}} N M \ln T}{C _ {\mathrm {l}} ^ {2}}\right), \tag {7}
$$
where  $\tilde{C}_i\leq O\left(\frac{1}{\bar{c}_i}\left(1 + \frac{1}{\bar{c}_i^2}\left(\sigma_i^2 +\frac{(C_{\mathrm{u}} - C_1)^2C_1}{C_{\mathrm{u}}}\right)\right)\right)$
$O\left(\frac{C_{\mathrm{u}}}{\bar{c}_i C_1}\right)$ . Furthermore, for any distribution  $\mathcal{D}$ , the regret is bounded as  $R_T = O\left(\frac{1}{C_1} \sqrt{C_{\mathrm{u}} N M T \ln T} + \frac{C_{\mathrm{u}}^2}{C_1^2} N M \ln T\right)$ .
The specific definition of  $\hat{C}_i$  is given in (19) in the appendix.
Remark 4.2. Bernstein-type confidence bounds used in (3) are essential to achieve the nearly optimal regret bound of  $R_{T} = O\left(\frac{1}{C_{1}} \sqrt{C_{\mathrm{u}} NMT \ln T}\right)$ . In fact, if we employ a standard confidence bound for  $\bar{c}_{i}$  instead of the Bernstein-type one given in (3), the parameter  $\sigma_{i}^{2}$  in the definition of  $\tilde{C}_{i}$  will be replaced with  $(C_{\mathrm{u}} - C_{1})^{2}$ , which leads a suboptimal regret bound of  $R_{T} = O\left(\sqrt{\frac{C_{\mathrm{u}}^{2}}{C_{1}^{3}} T \ln T}\right)$ .
From Proposition 2.2, the regret can be bounded as follows:
$$
R _ {T} \leq \mathbf {E} \left[ \sum_ {t = 1} ^ {T} \left(q ^ {\top} a ^ {*} - r _ {t} ^ {\top} a _ {t}\right) \right] + \frac {C _ {\mathrm {u}}}{C _ {1}} M \leq \frac {M}{C _ {1}} \mathbf {E} \left[ t _ {1} \right] + R _ {T} ^ {(1)} + R _ {T} ^ {(2)} + \frac {C _ {\mathrm {u}}}{C _ {1}} M, \tag {8}
$$
where we define
$$
R _ {T} ^ {(1)} = \mathbf {E} \left[ \sum_ {s = 1} ^ {S} \sum_ {t = t _ {s}} ^ {t _ {s + 1} - 1} \left(q ^ {\top} a ^ {*} - q ^ {\top} a _ {s} ^ {\prime}\right) \right], \quad R _ {T} ^ {(2)} = \mathbf {E} \left[ \sum_ {s = 1} ^ {S} \sum_ {t = t _ {s}} ^ {t _ {s + 1} - 1} \left(q ^ {\top} a _ {s} ^ {\prime} - r _ {t} ^ {\top} a _ {t}\right) \right]. \qquad (9)
$$
In these definitions of  $R_T^{(1)}$  and  $R_T^{(2)}$ , the index  $S$  represents the phase that includes the  $T$ -th round, and we define  $t_{S + 1} = T + 1$  exceptionally, for notational simplicity.
Let us next show the upper bounds for  $R_T^{(1)}$  and  $R_T^{(2)}$  separately. We will show that
$$
R _ {T} ^ {(1)} = O \left(M \sum_ {i \in \tilde {A}} \frac {\tilde {C} _ {i} \ln T}{\Delta_ {i}} + \frac {C _ {\mathrm {u}} ^ {2}}{C _ {1} ^ {2}} N M\right), \quad R _ {T} ^ {(2)} = O \left(\frac {C _ {\mathrm {u}} ^ {2}}{C _ {1} ^ {2}} N M \ln T\right). \tag {10}
$$
These, together with (8) and (5), prove the gap-dependent regret bound (7) in Theorem 4.1.
We here discuss an upper bound on  $R_T^{(1)}$  while the analysis of  $R_T^{(2)}$  is given in the supplementary material. We denote  $\tilde{A}_s = A_s' \backslash A^*$ . Define  $d_i(t)$  by  $d_i(t) = \sqrt{\frac{\tilde{C}_i}{\tilde{c}_i} \frac{\ln t}{T_i(t)}}$ , where  $\tilde{C}_i$  is defined by (19) in the appendix so that  $\tilde{C}_i = \Theta \left( \frac{1}{\tilde{c}_i} \left( 1 + \frac{1}{\tilde{c}_i^2} \left( \sigma_i^2 + \frac{(C_{\mathrm{u}} - C_1)^2 C_1}{C_{\mathrm{u}}} \right) \right) \right)$ . We then have  $\hat{q}_i(t) - q_i(t) \leq d_i(t)$  with high probability. In fact, we can show the following lemma using concentration inequalities, of which proof can be found in the appendix.
Lemma 4.3. For any  $s$ , with a probability at least  $1 - 6N / t_s^2$ , we have  $\Delta_{a_s'} \leq \sum_{i \in \tilde{A}_s} d_i(t_s)$ . Consequently, we have  $R_T^{(1)} \leq \mathbf{E}[\hat{R}_T] + O\left(\frac{MN}{C_1t_1}\right)$ , where we define  $\hat{R}_T = \sum_{s=1}^{S} l_s \Delta_{a_s'} \mathbf{1}[\mathcal{F}_s]$  with  $\mathcal{F}_s = \{\Delta_{a_s'} \leq \sum_{i \in \tilde{A}_s} d_i(t_s), \Delta_{a_s'} > 0\}$ .
We can show an upper bound on  $\hat{R}_T$  in a way similar to what Kveton et al. [25] did, as follows:
Lemma 4.4.  $\hat{R}_T$  defined in Lemma 4.3 is bounded as  $\mathbf{E}[\hat{R}_T] = O\left(M\sum_{i\in \tilde{A}}\frac{\tilde{C}_i\ln T}{\Delta_i}\right)$ .
This bound on  $\hat{R}_T$  together with Lemma 4.3 and (5) leads to the first part of (10).
4.1 Gap-Free Regret Bound
We can obtain the gap-free regret bound of  $R_{T} = \tilde{O}(\sqrt{\frac{C_{\mathrm{u}}}{C_{1}^{2}} NMT})$  in Theorem 4.1 by modifying the analysis of  $\hat{R}_{T}$ . For any  $\epsilon > 0$ ,  $\hat{R}_{T}$  can be bounded as  $\hat{R}_{T} \leq \sum_{s=1}^{S} (l_{s} \Delta_{a_{s}^{\prime}} \mathbf{1}[\mathcal{F}_{s}, \Delta_{a_{s}^{\prime}} > \epsilon] + l_{s} \Delta_{a_{s}^{\prime}} \mathbf{1}[\mathcal{F}_{s}, \Delta_{a_{s}^{\prime}} \leq \epsilon]) = O\left(\frac{C_{\mathrm{u}}}{C_{1}^{2}} MN \frac{\ln T}{\epsilon} + \epsilon T\right)$ , where the last inequality follows from the same argument for showing Lemma 4.4. By setting  $\epsilon = \sqrt{\frac{C_{\mathrm{u}}MN \ln T}{C_{1}^{2}T}}$ , we obtain  $\hat{R}_{T} = O\left(\frac{1}{C_{1}} \sqrt{C_{\mathrm{u}}MNT \ln T}\right)$ . From this, (8), the second part of (10), and Lemma 4.3, we obtain the gap-free regret bound presented in Theorem 4.1. This completes the proof of Theorem 4.1.
4.2 Regret Lower Bound
As shown in Theorem 4.1, the proposed algorithm achieves  $R_{T} = O\left(\frac{1}{C_{1}} \sqrt{C_{\mathrm{u}} NMT}\right)$ . This is tight up to a logarithmic factor in  $T$ . In fact, we have the following lower bound:
Theorem 4.5. For any  $N, M, T$  such that  $N \geq M$  and  $T = \Omega(C_{\mathrm{u}})$ , there exists a problem instance for which any algorithm suffers regret of  $R_T = \Omega(\frac{1}{C_1} \min \left\{\sqrt{C_{\mathrm{u}} NMT}, MT\right\})$ .
In the proof of this theorem, we first focus on the case of  $M = 1$ . If we consider the case of  $c_{ti} = C_1$ , our model is equivalent to the  $N$ -armed bandit with time horizon  $T' = \Theta (T / C_1)$ , which leads to a lower bound of  $\Omega (\sqrt{NT'}) = \Omega (\sqrt{NT / C_1})$  with the aid of the lower bound for multi-armed bandit problems given in Theorem 5.1 by Auer et al. [6]. When considering the case of  $r_{ti} = 1$  and  $c_{ti} \in \{C_1, C_u\}$ , we can show a regret lower bound of  $\Omega (\frac{C_u - C_1}{C_1C_u}\sqrt{C_uNT})$ , by using the proof technique by Badanidiyuru et al. [7]. Combining these two bounds, we obtain an  $\Omega (\frac{1}{C_1}\sqrt{C_uNT})$ -lower bound for the case of  $M = 1$ . From this result and the technique used in Proposition 2 by Kveton et al. [25], we have an  $\Omega (\frac{1}{C_1}\sqrt{C_uMNT})$ -lower bound for  $M > 1$ .
We conducted small experiments to evaluate the practical performance of the proposed algorithm using a synthetic dataset. We set the parameters for the dataset as follows:  $\mathcal{A} = \{A\subseteq [N]\mid$
Figure 2: Instance with small  $\Delta$ .
Figure 3: Instance with large  $\Delta$
$|A| \leq M\}$  with  $(N, M) = (4, 2)$ ,  $T = 10000$ ,  $C_{\mathrm{u}} = 6$ ,  $C_{\mathrm{l}} = 1$ ,  $\bar{r} = (0.5, 0.5, 0.5, 0.5)$ . For the expected processing times, we set  $\bar{c} = (1.5, 1.5, 2.0, 2.0)$  as a problem instance with small  $\Delta$  or  $\bar{c} = (1.5, 1.5, 5.0, 5.0)$  as a problem instance with large  $\Delta$ . Each  $c_{ti}$  and  $r_{ti}$  are generated so that  $c_{ti} - 1$  follows binomial distributions over  $\{0, 1, \dots, C_{\mathrm{u}} - 1\}$  and  $r_{ti}$  follows Bernoulli distributions over  $\{0, 1\}$ . For comparison, we have added the results of applying UCB-BV1 by Ding et al. [15] and CombUCB1 by Kveton et al. [24]. More detailed information on the experimental setup is provided in the supplementary material.
Figures 2 and 3 show the empirical means of regret computed by 100 repetitions of independent experiments. We depict 2-standard deviations of the empirical regret by the shaded areas. These results imply that the proposed algorithm performs better experimentally than other algorithms. We note that UCB-BV1 and CombUCB1 cannot avoid a linear regret as they choose a set in  $\mathcal{A}$  and wait for them all to be completed, which causes idling time with non-zero probability every time. It is also worth mentioning that the empirical performance of the proposed algorithm is much better than Theorem 4.1 predicts. In fact, under the parameter settings of these experiments, the values in Theorem 4.1 can be approximated as:  $\frac{1}{C_1} \sqrt{C_{\mathrm{u}} NMT \ln T} \approx 2100$  and  $\frac{C_{\mathrm{u}}}{C_1} N M \ln T \approx 2650$ .
One of the most relevant studies to this work would be those on combinatorial semi-bandits, in which a family  $F \subseteq 2^{[N]}$  of subsets of arms  $[N]$  is given and the player sequentially chooses  $A_{t} \in F$  and then observes the obtained rewards  $r_{ti}$  for each  $i \in A_{t}$ . Studies on combinatorial semi-bandits are classified into those on stochastic models [25, 24, 34], in which  $r_{t}$  are i.i.d. for  $t$ , and those on adversarial models [33, 5], in which  $\{r_t\}$  are arbitrary sequences that may be chosen in an adversarial manner. As noted in the introduction, the stochastic combinatorial semi-bandits are special cases of bandit task assignment. The stochastic combinatorial semi-bandits are known to admit sublinear regret. Typical approaches to achieve sublinear regret include upper confidence bounds (UCB)-type algorithms [25, 24] and Thompson sampling algorithms [34].
Our problem is also relevant to bandit problems with budget constraints. Ding et al. [15] consider a stochastic multi-armed bandit (MAB) problem with a budget constraint, in which the player observes the reward  $r_{ti_t}$  and the cost  $c_{ti_t}$  for the chosen arm  $i_t$  in each round  $t$ . The goal of the player is to maximize the cumulative rewards earned before cumulative costs reach a given total budget amount. This can be seen as a special case of our problem. As shown in Ding et al. [15], in the budget-constrained MAB problem, the reward of the optimal policy can be characterized by the expected reward-to-cost. This fact is generalized to our setting in Proposition 2.2, which will be used in our regret analysis. It should be mentioned that the budget-constrained MAB problem has been generalized to bandits with multiple plays [37], which differs from our model, as problems with processing times cannot be reduced to budget-constrained models in general. In addition to their work, there are various generalizations and variants, such as contextual bandit models [36], models with general distributions [11, 12], and models with multiple budgets [2, 31, 7].
While delayed feedback models [13, 38, 3] are similar to our model in that rewards earned will be revealed in later rounds, there are some essential differences. For example, in our model, the selected arm is occupied during the processing time and cannot be selected, while in the delayed feedback model, the arm can be selected while waiting for feedback. The former model can handle situations
where the arm corresponds to a labor or computing resource, as in the examples shown in Examples 1 and 2.
Phased-update or lazy-update approaches were incorporated in other bandit problems and online learning problems such as a phased version of UCB for MAB (e.g., Exercise 7.5 of the book [26]) and the rarely switching OFUL algorithm for linear stochastic bandits [1]. The motivation for using phased-update approaches is primarily to reduce the computational cost [1, 22], to minimize switching cost [2, 21], or to address batched settings [29, 19]. To our knowledge, our proposed algorithm is the first algorithm that applies such a phased-update approach to combinatorial semi-bandits. It allows us to reduce the number of oracle calls, and more importantly, to achieve the nearly tight regret bounds for the bandit task assignment. Our study reveals new usefulness of a phased-update approach.
Another line of research related to our problem is the online matching or online allocation problems with reusable resources [14, 16, 32]. The problem is formalized with a bipartite graph, and we choose edges for sequentially arriving vertices. The model is similar to ours in that each chosen edge will be unavailable for a certain period of time that follows a probability distribution. However, it is different in that the bipartite graph is unknown at first and all the distributions are given. Moreover, those papers adopt competitive ratio analyses rather than regret analyses, i.e., they evaluate the performance compared to a stronger agent who knows the sequence of arriving vertices. Thus, the existing papers are incomparable with our work.
As mentioned in the introduction, the problem setting of blocking bandit [8] appears to be similar to ours while the difficulties are significantly different. Considering unconstrained situations can highlight the differences in the problem. In the task assignment problem with no constraints, the optimal strategy will be trivial, in which we make each task always processing, i.e., in each round, we start every task that is completed at that round. On the other hand, as the reviewer commented, blocking bandits only allow us to play a single action in each round, which makes the optimal strategy nontrivial. In fact, finding the optimal policy can be an NP-hard problem even if the true distributions of  $r$  and  $c$  are given. This is an illustrative example of why bandit task assignment can be easier than blocking bandits.
7 Conclusion and limitation
In this paper, we introduced a new problem setting that we referred to as bandit task assignment, for which we proposed an algorithm with regret bounds. A limitation of this work is that we need to know the upper and lower bounds  $C_{\mathrm{u}}$ ,  $C_1$  on the processing times, which may not be possible to assume in some real-world applications. To remove this assumption, it would be necessary to modify the framework of the problem. One possible modification would be to allow the player to suspend processing tasks. In this modified problem setup, even in cases where  $C_{\mathrm{u}}$  is incorrectly underestimated, it will be possible to immediately abort the task and correct  $C_{\mathrm{u}}$  when the underestimation is detected. We expect that this strategy and the appropriate parameter update rules for  $C_{\mathrm{u}}$  and  $C_1$  allow us to remove the assumption of prior knowledge about processing times. Another future direction is to extend the problem to a decentralized setting, in which multiple agents collaborate to make task assignment decisions. We believe that such an extension of the problem provides insight into real-world applications where multiple entities perform task assignments, such as crowdsourcing and federated learning.
We deeply appreciate many comprehensive comments and feedback by anonymous reviewers. HS was supported by JSPS KAKENHI Grant Numbers JP17K12646, JP21K17708, and JP21H03397, Japan. TF was supported by JSPS KAKENHI Grant Numbers JP20H05965, JP21K11759, and JP21H03397, Japan. NK was supported by JSPS KAKENHI Grant Numbers JP22H05001, JP20H05795, and JP21H03397, Japan. KK was supported by JSPS KAKENHI Grant Numbers JP22H05001 and JP20A402, Japan.
Lemma A.1 (Empirical bernstein inequalities, [4]). Let  $X_{1}, X_{2}, \ldots, X_{n} \in [C_{1}, C_{\mathrm{u}}]$  be a sequence of i.i.d. random variables with mean  $\mu$  and variance  $\sigma^2$ . Let  $\hat{\mu}_n = \sum_{i=1}^{n} X_i / n$  and  $V_n = \sum_{i=1}^{n} (X_i - \hat{\mu}_n)^2 / n$ . For any  $\delta \in (0,1)$ , with probability  $1 - 4\delta$ , we have
$$
\left| \hat {\mu} _ {n} - \mu \right| \leq \sqrt {\frac {V _ {n}}{n} \ln \frac {1}{\delta}} + \frac {3 \left(C _ {\mathrm {u}} - C _ {\mathrm {l}}\right)}{n} \ln \frac {1}{\delta} \leq \sqrt {\frac {\sigma^ {2}}{n} \ln \frac {1}{\delta}} + \frac {5 \left(C _ {\mathrm {u}} - C _ {\mathrm {l}}\right)}{n} \ln \frac {1}{\delta}. \tag {11}
$$
Proof. Theorem 1 by Audibert et al. [4] implies that the first inequality holds with probability  $1 - 3\delta$ . The second inequality follows from Bernstein's inequality for  $V_{t}$ :
$$
\Pr \left[ V _ {t} \geq \sigma^ {2} + \sqrt {\frac {2 \left(C _ {\mathrm {u}} - C _ {\mathrm {l}}\right) ^ {2} \sigma^ {2}}{n} \ln \frac {1}{\delta}} + \frac {2 \left(C _ {\mathrm {u}} - C _ {\mathrm {l}}\right) ^ {2}}{3 n} \ln \frac {1}{\delta} \right] \leq \delta . \tag {12}
$$
From this, with probability  $1 - \delta$ , we have
$$
\begin{array}{l} \sqrt {\frac {V _ {n}}{n} \ln \frac {1}{\delta}} \leq \sqrt {\frac {1}{n} \left(\sigma^ {2} + \sqrt {\frac {2 (C _ {\mathrm {u}} - C _ {\mathrm {l}}) ^ {2} \sigma^ {2}}{n} \ln \frac {1}{\delta}} + \frac {2 (C _ {\mathrm {u}} - C _ {\mathrm {l}}) ^ {2}}{3 n} \ln \frac {1}{\delta}\right) \ln \frac {1}{\delta}} \\ = \sqrt {\frac {\sigma^ {2}}{n} \ln \frac {1}{\delta}} \cdot \sqrt {1 + \sqrt {\frac {2 (C _ {\mathrm {u}} - C _ {\mathrm {l}}) ^ {2}}{\sigma^ {2} n} \ln \frac {1}{\delta}} + \frac {2 (C _ {\mathrm {u}} - C _ {\mathrm {l}}) ^ {2}}{3 \sigma^ {2} n} \ln \frac {1}{\delta}}} \\ \leq \sqrt {\frac {\sigma^ {2}}{n} \ln \frac {1}{\delta}} \cdot \left(1 + 2 \sqrt {\frac {(C _ {\mathrm {u}} - C _ {\mathrm {l}}) ^ {2}}{\sigma^ {2} n} \ln \frac {1}{\delta}}\right) \leq \sqrt {\frac {\sigma^ {2}}{n} \ln \frac {1}{\delta}} + \frac {2 (C _ {\mathrm {u}} - C _ {\mathrm {l}})}{n} \ln \frac {1}{\delta}, \\ \end{array}
$$
where the second inequality follows from  $\sqrt{1 + x + y} \leq \sqrt{1 + x} + \sqrt{y} \leq 1 + x / 2 + \sqrt{y}$  that holds for any  $x, y \geq 0$ . This implies that the second inequality of (11) holds with probability  $1 - \delta$ .
A.1 Proof of Proposition 2.2
Proof. For  $t = 1,2,\ldots ,T$ , denote  $a_{t} = \pi_{t}(h_{t})$  and define  $b_{t}$  by (1). For  $t > T$ , let  $a_{t} = 0$  for the notational simplicity. Then, as  $r_t$  is independent of  $a_{t}$ , we have
$$
\mathbf {E} \left[ \sum_ {t = 1} ^ {T} r _ {t} ^ {\top} a _ {t} \right] = \mathbf {E} \left[ \sum_ {t = 1} ^ {T} \bar {r} ^ {\top} a _ {t} \right] = \bar {r} ^ {\top} \mathbf {E} \left[ \sum_ {t = 1} ^ {T} a _ {t} \right]. \tag {13}
$$