SlowGuess/NeurIPS_2023_OCR huggingface dataset:
From (1), we have  $\mathbf{E}\left[\sum_{t=1}^{T+C_{\mathrm{u}}} (a_{ti} + b_{ti})\right] = \mathbf{E}\left[\sum_{t=1}^{T+C_{\mathrm{u}}}\left(a_{ti} + \sum_{s=1}^{t-1} a_{si} \mathbf{1}[s + c_{si} \geq t]\right)\right] = \mathbf{E}\left[\sum_{t=1}^{T} a_{ti} c_{ti}\right] = \bar{c}_{i} \mathbf{E}\left[\sum_{t=1}^{T} a_{ti}\right]$  for any  $i$ , where the last equality follows from the fact that  $c_{t}$  is independent of  $a_{t}$ . Hence, we have  $\mathbf{E}\left[\sum_{t=1}^{T+C_{\mathrm{u}}} (a_{t} + b_{t})\right] = D \mathbf{E}\left[\sum_{t=1}^{T} a_{t}\right]$ , where  $D \in \mathbb{R}_{>0}^{N \times N}$  denote the diagonal matrix with entries  $\bar{c}_{1}, \bar{c}_{2}, \ldots, \bar{c}_{N}$ . As we have  $a_{t} + b_{t} \in \mathcal{A}$ , this implies that  $\frac{1}{T+C_{\mathrm{u}}} D \mathbf{E}\left[\sum_{t=1}^{T} a_{t}\right] = \mathbf{E}\left[\frac{1}{T+C_{\mathrm{u}}} \sum_{t=1}^{T+C_{\mathrm{u}}} (a_{t} + b_{t})\right]$  is in the convex hull of  $\mathcal{A}$ . We hence have  $\bar{r}^{\top} \mathbf{E}\left[\sum_{t=1}^{T} a_{t}\right] = (T + C_{\mathrm{u}})(D^{-1} \bar{r})^{\top}\left(\frac{1}{T+C_{\mathrm{u}}} D \mathbf{E}\left[\sum_{t=1}^{T} a_{t}\right]\right) \leq (T + C_{\mathrm{u}}) \max_{a \in \mathcal{A}}\left\{q^{\top} a\right\}$ , where the last inequality follows from  $D^{-1} \bar{r} = q$  and the fact that  $\frac{1}{T+C_{\mathrm{u}}} D \mathbf{E}\left[\sum_{t=1}^{T} a_{t}\right]$  is in the convex hull of  $\mathcal{A}$ . Combining this with (13), we obtain the bound in Proposition 2.2.
A.2 Proof of Lemma 3.1
Proof. From (5), we have  $\min_{i\in A_s'}T_i(t_s)\geq C_\mathrm{u} / C_1$ , which implies  $l_{s} = C_{1}\min_{i\in A_{s}^{\prime}}T_{i}(t_{s}) + 2C_{\mathrm{u}}\geq$ $2C_{\mathrm{u}}$ . From this, for all  $i\in A_s'$  we have  $T_{i}(t_{s + 1}) - T_{i}(t_{s})\geq (l_{s} - 2C_{\mathrm{u}}) / C_{\mathrm{u}}\geq l_{s} / (2C_{\mathrm{u}})$ . This means the first inequality in Lemma 3.1 holds. Further, we have
$$
\begin{array}{l} T_{i}(t_{s + 1}) - T_{i}(t_{s})\leq (l_{s} + 1) / C_{1}\leq (C_{1}\min_{i\in A_{s}^{\prime}}T_{i}(t_{s}) + C_{\mathrm{u}} + 1) / C_{1} \\ = \min _ {i \in A _ {s} ^ {\prime}} T _ {i} (t _ {s}) + C _ {\mathrm {u}} / C _ {\mathrm {l}} + 1 / C _ {\mathrm {l}} \leq 3 \min _ {i \in A _ {s} ^ {\prime}} T _ {i} (t _ {s}), \\ \end{array}
$$
which implies that the second inequality in Lemma 3.1 holds.
A.3 Proof of Lemma 3.2
Proof. Let  $i^{*} \in A_{s}^{\prime}$  be a task such that  $T_{i^{*}}(s) = \min_{i \in A_{s}^{\prime}} T_{i}(t_{s})$ . We then have  $l_{s} = C_{1}T_{i^{*}}(s) + 2C_{\mathrm{u}}$ . As the task  $i^{*}$  is completed at least  $(l_{s} - 2C_{\mathrm{u}}) / C_{\mathrm{u}} = (C_{\mathrm{l}} / C_{\mathrm{u}})T_{i^{*}}(s)$  times within the  $s$ -th phase, we have  $T_{i^{*}}(s + 1) - T_{i^{*}}(s) \geq (C_{\mathrm{l}} / C_{\mathrm{u}})T_{i^{*}}(s)$ , which implies  $T_{i^{*}}(s + 1) \geq (1 + C_{\mathrm{l}} / C_{\mathrm{u}})T_{i^{*}}(s)$ . Further, if  $i \in A_{s^{\prime}}^{\prime}$  for some  $s^{\prime} < s$ , we have  $T_{i}^{\prime}(s) \geq C_{\mathrm{u}} / C_{\mathrm{l}}$  as shown in the proof of Lemma 3.1. Hence, if  $s \geq NK + 1$  for some positive integer  $K$ , there exists  $i \in [N]$  such that  $T_{t_i}(s) \geq (C_{\mathrm{u}} / C_{\mathrm{l}})(1 + C_{\mathrm{l}} / C_{\mathrm{u}})^{K - 1}$ . From this, as  $t_s \geq C_1T_i(t_s)$  holds for any  $i \in [N]$ , we have  $t_s \geq C_1(1 + C_1 / C_\mathrm{u})^{s / N - 2}$ .
A.4 Proof of Lemma 4.3
Let us first show that  $\Delta_{a_s'} \leq \sum_{i \in \tilde{A}_s} d_i(t_s)$  holds with a probability of at least  $1 - 6N / t_s^2$ . Fix  $t$  arbitrarily.
For any  $i$  and let  $\tau(1) < \tau(2) < \dots$  represent all the indices of rounds at which the  $i$ -th task is started. Define the sequence  $E_1, E_2, \ldots$  by  $E_j = \sum_{s=1}^j (r_i(\tau(s)) - \bar{r}_i)$ . Then, since  $E_j$  is a sum of  $j$  i.i.d. random variables, the Azuma-Hoeffding inequality implies  $\operatorname*{Pr}[|E_j| \geq \sqrt{1.5j \cdot \ln t}] \leq \frac{2}{t}$  for any  $t$  and  $j$ . We hence have
$$
\begin{array}{l} \operatorname * {P r} \left[ | \hat {r} _ {i} (t) - \bar {r} _ {i} | \geq d _ {i} ^ {r} (t) \right] = \operatorname * {P r} \left[ | E _ {T _ {i} (t)} | \geq \sqrt {1 . 5 T _ {i} (t) \cdot \ln t} \right] \\ \leq \Pr [ \exists j \in [ t ], \quad | E _ {j} | \geq \sqrt {1 . 5 j \cdot \ln t} ] \leq \sum_ {j = 1} ^ {t} \Pr [ | E _ {j} | \geq \sqrt {1 . 5 j \cdot \ln t} ] \leq \sum_ {j = 1} ^ {t} \frac {2}{t ^ {3}} \leq \frac {2}{t ^ {2}}. \tag {14} \\ \end{array}
$$
which means that the first part of (4) holds. Similarly, by considering  $E_{j} = \sum_{s = 1}^{j}(c_{i}(\tau (s)) - \bar{c}_{i})$ . it can be shown from Lemma A.1 that
$$
\Pr \left[ | \hat {c} _ {i} (t) - \bar {c} _ {i} | \leq d _ {i} ^ {c} (t) \leq \sqrt {\frac {3 \sigma_ {i} ^ {2} \ln t}{T _ {i} (t)}} + \frac {1 5 \left(C _ {\mathrm {u}} - C _ {\mathrm {l}}\right) \ln t}{T _ {i} (t)} \right] \geq 1 - \frac {4}{t ^ {2}}. \tag {15}
$$
$$
\begin{array}{l} \sqrt {\frac {3 \sigma_ {i} ^ {2} \ln t}{T _ {i} (t)}} + \frac {1 5 \left(C _ {\mathrm {u}} - C _ {\mathrm {l}}\right) \ln t}{T _ {i} (t)} \leq \left(\sqrt {3 \sigma_ {i} ^ {2}} + 1 5 \left(C _ {\mathrm {u}} - C _ {\mathrm {l}}\right) \sqrt {\frac {C _ {\mathrm {l}}}{9 0 C _ {\mathrm {u}}}}\right) \sqrt {\frac {\ln t}{T _ {i} (t)}} \\ \leq \sqrt {3 \sigma_ {i} ^ {2}} \sqrt {\frac {C _ {1}}{9 0 C _ {\mathrm {u}}}} + 1 5 \left(C _ {\mathrm {u}} - C _ {1}\right) \frac {C _ {1}}{9 0 C _ {\mathrm {u}}} \leq \sqrt {\left(C _ {\mathrm {u}} - C _ {1}\right) \left(\bar {c} _ {i} - C _ {1}\right)} \sqrt {\frac {C _ {1}}{3 0 C _ {\mathrm {u}}}} + \left(C _ {\mathrm {u}} - C _ {1}\right) \frac {C _ {1}}{6 C _ {\mathrm {u}}} \\ \leq \frac {\sqrt {(\bar {c} _ {i} - C _ {1}) C _ {1}}}{\sqrt {3 0}} + \frac {C _ {1}}{6} \leq \frac {\bar {c} _ {i}}{4}, \tag {16} \\ \end{array}
$$
where the first and second inequalities follow from the lower bound on  $T_{i}(t_{s})$  in (5), and the third and last inequalities follow from  $C_1\leq \bar{c}_i\leq C_\mathrm{u}$ . Hence, with probability  $1 - 6N / t^2$
$$
\hat {q} _ {i} (t) = \frac {\operatorname* {m i n} \{1 , \hat {r} _ {i} (t) + d _ {i} ^ {r} (t) \}}{\operatorname* {m a x} \left\{C _ {1} , \hat {c} _ {i} (t) - d _ {i} ^ {c} (t) \right\}} \geq \frac {\bar {r} _ {i}}{\bar {c} _ {i}} = q _ {i}. \tag {17}
$$
As (16) implies  $\bar{c}_i - 2d_i^c (t)\geq \bar{c}_i / 2$  , we have
$$
\begin{array}{l} \hat {q} _ {i} (t) = \frac {\min \{1 , \hat {r} _ {i} (t) + d _ {i} ^ {r} (t) \}}{\max \{C _ {1} , \hat {c} _ {i} (t) - d _ {i} ^ {c} (t) \}} \leq \frac {\min \{1 , \bar {r} _ {i} + 2 d _ {i} ^ {r} (t) \}}{\max \{C _ {1} , \bar {c} _ {i} - 2 d _ {i} ^ {c} (t) \}} =: \frac {\tilde {r} _ {i}}{\tilde {c} _ {i}} \\ = \frac {\bar {r} _ {i}}{\bar {c} _ {i}} + \frac {\tilde {r} _ {i} - \bar {r} _ {i}}{\bar {c} _ {i}} + \frac {\tilde {r} _ {i} (\bar {c} _ {i} - \tilde {c} _ {i})}{\bar {c} _ {i} \tilde {c} _ {i}} \leq q _ {i} + \frac {2 d _ {i} ^ {r} (t)}{\bar {c} _ {i}} + \frac {2 d _ {i} ^ {c} (t)}{\bar {c} _ {i} \tilde {c} _ {i}} \leq q _ {i} + \frac {2 d _ {i} ^ {r} (t)}{\bar {c} _ {i}} + \frac {4 d _ {i} ^ {c} (t)}{\bar {c} _ {i} ^ {2}} \\ \leq q _ {i} + \left(\frac {2 \sqrt {1 . 5}}{\bar {c} _ {i}} + \frac {4}{\bar {c} _ {i} ^ {2}} \left(\sqrt {3 \sigma_ {i} ^ {2}} + 1 5 \left(C _ {\mathrm {u}} - C _ {\mathrm {l}}\right) \sqrt {\frac {C _ {\mathrm {l}}}{9 0 C _ {\mathrm {u}}}}\right)\right) \sqrt {\frac {\ln t _ {s}}{T _ {i} \left(t _ {s}\right)}} \\ = q _ {i} + \sqrt {\frac {\tilde {C} _ {i}}{\bar {c} _ {i}} \frac {\ln t _ {s}}{T _ {i} \left(t _ {s}\right)}} = q _ {i} + d _ {i} \left(t _ {s}\right), \tag {18} \\ \end{array}
$$
where we denote  $\tilde{r}_i = \min \{1, \bar{r}_i + 2d_i^r(t)\}$  and  $\tilde{c}_i = \max \{C_1, \bar{c}_i - 2d_i^c(t)\} \geq \bar{c}_i / 2$ . We here defined  $\tilde{C}_i$  by
$$
\tilde {C} _ {i} = \frac {1}{\bar {c} _ {i}} \left(2 \sqrt {1 . 5} + \frac {4}{\bar {c} _ {i}} \left(\sqrt {3 \sigma_ {i} ^ {2}} + 1 5 \left(C _ {\mathrm {u}} - C _ {\mathrm {l}}\right) \sqrt {\frac {C _ {\mathrm {l}}}{9 0 C _ {\mathrm {u}}}}\right)\right) ^ {2}. \tag {19}
$$
We then have
$$
\begin{array}{l} \Delta_ {a _ {s} ^ {\prime}} = \sum_ {i \in A ^ {*} \backslash A _ {s} ^ {\prime}} q _ {i} - \sum_ {i \in A _ {s} ^ {\prime} \backslash A ^ {*}} q _ {i} \leq \sum_ {i \in A ^ {*} \backslash A _ {s} ^ {\prime}} \hat {q} _ {i} (t _ {s}) - \sum_ {i \in A _ {s} ^ {\prime} \backslash A ^ {*}} q _ {i} \\ \leq \sum_ {i \in A _ {s} ^ {\prime} \backslash A ^ {*}} \hat {q} _ {i} \left(t _ {s}\right) - \sum_ {i \in A _ {s} ^ {\prime} \backslash A ^ {*}} q _ {i} \leq \sum_ {i \in A _ {s} ^ {\prime} \backslash A ^ {*}} d _ {i} \left(t _ {s}\right) = \sum_ {i \in \tilde {A} _ {s}} d _ {i} \left(t _ {s}\right), \tag {20} \\ \end{array}
$$
where the first and last inequalities follow from (17) and (18), respectively, and the second inequality follows from the definition of  $a_s'$ . Indeed,  $a_s' \in \arg \max_{a \in \mathcal{A}} \hat{q}(t_s)^{\top} a$  implies the following inequality:  $\sum_{i \in A^* \setminus A_s'} \hat{q}_i(t_s) - \sum_{i \in A_s' \setminus A^*} \hat{q}_i(t_s) = \hat{q}(t_s)^{\top} a^* - \hat{q}(t_s)^{\top} a_s' \leq 0$ .
Let us next show the bound on  $R_T^{(1)}$ . We decompose  $R_T^{(1)}$  as follows:
$$
R _ {T} ^ {(1)} = \mathbf {E} \left[ \sum_ {s = 1} ^ {S} l _ {s} \Delta_ {a _ {s} ^ {\prime}} \right] = \mathbf {E} \left[ \sum_ {s = 1} ^ {S} l _ {s} \Delta_ {a _ {s} ^ {\prime}} \mathbf {1} \left[ \mathcal {F} _ {s} \right] + \sum_ {s = 1} ^ {S} l _ {s} \Delta_ {a _ {s} ^ {\prime}} \mathbf {1} \left[ \bar {\mathcal {F}} _ {s} \right] \right] = \mathbf {E} \left[ \hat {R} _ {T} + \sum_ {s = 1} ^ {S} l _ {s} \Delta_ {a _ {s} ^ {\prime}} \mathbf {1} \left[ \bar {\mathcal {F}} _ {s} \right] \right], \tag {21}
$$
where  $\bar{\mathcal{F}}_s$  denotes the complement of the event  $\mathcal{F}_s$ . The second part of RHS can be bounded as
$$
\begin{array}{l} \mathbf {E} \left[ \sum_ {s = 1} ^ {S} l _ {s} \Delta_ {a _ {s} ^ {\prime}} \mathbf {1} [ \bar {\mathcal {F}} _ {s} ] \right] = \mathbf {E} \left[ \sum_ {s = 1} ^ {S} l _ {s} \Delta_ {a _ {s} ^ {\prime}} \mathbf {1} \left[ \Delta_ {a _ {s} ^ {\prime}} > \sum_ {i \in \tilde {A} _ {s}} d _ {i} (t _ {s}) \right] \right] \\ \leq \mathbf {E} \left[ \sum_ {s = 1} ^ {S} l _ {s} \frac {M}{C _ {1}} \frac {6 N}{t _ {s} ^ {2}} \right] = \frac {6 M N}{C _ {1}} \mathbf {E} \left[ \sum_ {s = 1} ^ {S} \frac {l _ {s}}{t _ {s} ^ {2}} \right] \tag {22} \\ \end{array}
$$
where the first inequality follows from the fact that (20) holds with a probability of at least  $1 - 6N / t_s^2$  and the inequality of  $\Delta_{a_s'} \leq M / C_1$ . We have
$$
\frac {l _ {s}}{t _ {s} ^ {2}} = \frac {t _ {s + 1} - t _ {s}}{t _ {s} ^ {2}} = \frac {t _ {s + 1} - t _ {s}}{t _ {s + 1} t _ {s}} \frac {t _ {s + 1}}{t _ {s}} = \left(\frac {1}{t _ {s}} - \frac {1}{t _ {s + 1}}\right) \frac {t _ {s + 1}}{t _ {s}} \leq 5 \left(\frac {1}{t _ {s}} - \frac {1}{t _ {s + 1}}\right),
$$
$$
\frac {t _ {s + 1}}{t _ {s}} = 1 + \frac {l _ {s}}{t _ {s}} \leq 1 + \frac {1}{t _ {s}} \left(C _ {1} \max  _ {i \in [ N ]} T _ {i} ^ {\prime} (s) + 2 C _ {\mathrm {u}}\right) \leq 1 + \frac {3 C _ {1}}{t _ {s}} \max  _ {i \in [ N ]} T _ {i} ^ {\prime} (s) \leq 1 + \frac {3 C _ {1}}{t _ {s}} \left(\frac {t _ {s}}{C _ {1}} + 1\right) \leq 5.
$$
$$
\sum_ {s = 1} ^ {S} \frac {l _ {s}}{t _ {s} ^ {2}} \leq 5 \sum_ {s = 1} ^ {S} \left(\frac {1}{t _ {s}} - \frac {1}{t _ {s + 1}}\right) \leq 5 \frac {1}{t _ {1}}. \tag {23}
$$
Combining (21), (22) and (23), we obtain the bound on  $R_T^{(1)}$  in Lemma 4.3.
A.5 Proof of Lemma A.2
The following lemma can be shown in a way similar to that of Kveton et al. [25, Lemma 3].
Lemma A.2. Let  $\{\alpha_k\}$  and  $\{\beta_k\}$  be positive real sequences for which  $1 = \beta_0 > \beta_1 > \dots > \beta_k > \dots > 0$ ,  $\alpha_1 > \alpha_2 > \dots > \alpha_k > \dots > 0$ ,  $\lim_{k \to \infty} \alpha_k = \lim_{k \to \infty} \beta_k = 0$ , and  $\sum_{k=1}^{\infty} \frac{\beta_{k-1} - \beta_k}{\sqrt{\alpha_k}} \leq 1$ .
Suppose that  $\mathcal{F}_s$  occurs. There then exists  $k$  such that  $S_{s,k} = \{i\in \tilde{A}_s\mid \Delta_{a_s'}\leq M\sqrt{\alpha_k} d_i(t_s)\}$  satisfies  $|S_{s,k}|\geq \beta_kM$
Proof. Let us show the claim via proof by contradiction. Suppose that  $|S_{s,k}| < \beta_k M$  holds for all  $k$ . We then have
$$
\begin{array}{l} \sum_ {i \in \tilde {A} _ {s}} d _ {i} (t _ {s}) <   \sum_ {k = 1} ^ {\infty} \sum_ {i \in S _ {s, k - 1} \backslash S _ {s, k}} \frac {\Delta_ {a _ {s} ^ {\prime}}}{\tilde {C} M \sqrt {\alpha_ {k}}} = \frac {\Delta_ {a _ {s} ^ {\prime}}}{\tilde {C} M} \sum_ {k = 1} ^ {\infty} \frac {| S _ {s , k - 1} | - | S _ {s , k} |}{\sqrt {\alpha_ {k}}} \\ = \frac {\Delta_ {a _ {s} ^ {\prime}}}{\tilde {C} M} \left(\frac {| S _ {s , 0} |}{\sqrt {\alpha_ {1}}} + \sum_ {k = 1} ^ {\infty} | S _ {k} | \left(\frac {1}{\sqrt {\alpha_ {k + 1}}} - \frac {1}{\sqrt {\alpha_ {k}}}\right)\right) \\ <   \frac {\Delta_ {a _ {s} ^ {\prime}}}{\tilde {C}} \left(\frac {\beta_ {0}}{\sqrt {\alpha_ {1}}} + \sum_ {k = 1} ^ {\infty} \beta_ {k} \left(\frac {1}{\sqrt {\alpha_ {k + 1}}} - \frac {1}{\sqrt {\alpha_ {k}}}\right)\right) \\ = \frac {\Delta_ {a _ {s} ^ {\prime}}}{\tilde {C}} \sum_ {k = 1} ^ {\infty} \frac {\beta_ {k - 1} - \beta_ {k}}{\sqrt {\alpha_ {k}}} \leq \frac {\Delta_ {a _ {s} ^ {\prime}}}{\tilde {C}}, \\ \end{array}
$$
where the first inequality follows from the definition of  $S_{s,k}$ , i.e.,  $\tilde{CM}\sqrt{\alpha_k} d_i(t_s) < \Delta_{a_s'}$  holds for all  $i\in [N]\setminus S_{s,k}$ , and the second inequality follows from the initial assumption made in this proof. This means that  $\mathcal{F}_s$  occurs. Hence, if  $\mathcal{F}_s$  occurs, this initial assumption is false, which means that there exists  $k$  such that  $|S_{s,k}|\geq \beta_kM$ .
We will apply Lemma A.2 with  $\{\alpha_k\}$  and  $\{\beta_k\}$  such that  $\sum_{k=1}^{\infty} \frac{\alpha_k}{\beta_k} = O(1)$ , which are provided in Appendix A.4 in [25]. From Lemma A.2, if  $\mathcal{F}_s$  occurs,  $\sum_{i \in \tilde{A}} \mathbf{1}[i \in \tilde{A}_s, \Delta_{a_s'} \leq 2\tilde{C}M\sqrt{\alpha_k} d_i(t_s)] \geq \beta_k M$  holds for some  $k$ , which implies
$$
\mathbf {1} \left[ \mathcal {F} _ {s} \right] \leq \sum_ {k = 1} ^ {\infty} \frac {1}{\beta_ {k} M} \sum_ {i \in \tilde {A}} \mathbf {1} \left[ \mathcal {G} _ {s, k, i} \right], \quad \text {w h e r e} \quad \mathcal {G} _ {s, k, i} = \left\{i \in \tilde {A} _ {s}, \Delta_ {a _ {s} ^ {\prime}} \leq M \sqrt {\alpha_ {k}} d _ {i} \left(t _ {s}\right) \right\}. \tag {24}
$$
Proof. Using (24), we can bound  $\hat{R}_T$  as
$$
\begin{array}{l} \hat {R} _ {T} \leq \sum_ {i \in \tilde {A}} \sum_ {k = 1} ^ {\infty} \frac {1}{\beta_ {k} M} \sum_ {s = 1} ^ {S} l _ {s} \Delta_ {a _ {s} ^ {\prime}} \mathbf {1} \left[ \mathcal {G} _ {s, k, i} \right] \leq \sum_ {i \in \tilde {A}} \sum_ {k = 1} ^ {\infty} \frac {\sqrt {\alpha_ {k}}}{\beta_ {k}} \sum_ {s = 1} ^ {S} l _ {s} d _ {i} (t _ {s}) \mathbf {1} \left[ \mathcal {G} _ {s, k, i} \right] \\ = \sum_ {i \in \tilde {A}} \sum_ {k = 1} ^ {\infty} \frac {\sqrt {\alpha_ {k}}}{\beta_ {k}} \sum_ {s = 1} ^ {S} l _ {s} \sqrt {\frac {\tilde {C} _ {i}}{\bar {c} _ {i}} \frac {\ln t _ {s}}{T _ {i} \left(t _ {s}\right)}} \mathbf {1} \left[ \mathcal {G} _ {s, k, i} \right] \leq \sqrt {\ln T} \sum_ {i \in \tilde {A}} \sqrt {\frac {\tilde {C} _ {i}}{\bar {c} _ {i}}} \sum_ {k = 1} ^ {\infty} \frac {\sqrt {\alpha_ {k}}}{\beta_ {k}} \sum_ {s = 1} ^ {S} \frac {l _ {s} \mathbf {1} \left[ \mathcal {G} _ {s , k , i} \right]}{\sqrt {T _ {i} \left(t _ {s}\right)}}. \tag {25} \\ \end{array}
$$
Let us evaluate  $\sum_{s=1}^{S} \frac{l_s \mathbf{1}[\mathcal{G}_{s,k,i}]}{\sqrt{T_i(t_s)}}$  for some fixed  $k$  and  $i$ . If  $\mathcal{G}_{s,k,i}$  occurs, we have  $\Delta_i \leq \Delta_{a_s'} \leq M \sqrt{\alpha_k} d_i(t_s) \leq M \sqrt{\frac{\alpha_k \tilde{C}_i \ln T}{\tilde{c}_i T_i(t_s)}}$  which implies that  $\sqrt{T_i(t_s)} \leq M \sqrt{\frac{\alpha_k \tilde{C}_i \ln T}{\tilde{c}_i}} \frac{1}{\Delta_i} =: \gamma$ . We hence have
$$
\sum_ {s = 1} ^ {S} \frac {l _ {s} \mathbf {1} [ \mathcal {G} _ {s , k , i} ]}{\sqrt {T _ {i} \left(t _ {s}\right)}} \leq \sum_ {s = 1} ^ {S} \frac {l _ {s} \mathbf {1} [ i \in \tilde {A} _ {s} ] \mathbf {1} [ \sqrt {T _ {i} \left(t _ {s}\right)} \leq \gamma ]}{\sqrt {T _ {i} \left(t _ {s}\right)}}. \tag {26}
$$
The expectation of the right-hand side can be bounded as follows. From Lemma A.4 in the Appendix, we have
$$
\mathbf {E} \left[ \frac {l _ {s} \mathbf {1} [ i \in \tilde {A} _ {s} ]}{\sqrt {T _ {i} (t _ {s})}} \right] \leq 2 \bar {c} _ {i} \mathbf {E} \left[ \frac {T _ {i} \left(t _ {s + 1}\right) - T _ {i} \left(t _ {s}\right)}{\sqrt {T _ {i} (t _ {s})}} \right] \tag {27}
$$
By using Lemma 3.1, we obtain  $\frac{T_i(t_{s+1}) - T_i(t_s)}{\sqrt{T_i(t_s)}} = 3\frac{T_i(t_{s+1}) - T_i(t_s)}{\sqrt{4T_i(t_s)} + \sqrt{T_i(t_s)}} \leq 3\frac{T_i(t_{s+1}) - T_i(t_s)}{\sqrt{T_i(t_{s+1})} + \sqrt{T_i(t_s)}} = 3(\sqrt{T_i(t_{s+1})} - \sqrt{T_i(t_s)})$ . Combining this with (26) and (27), we obtain  $\mathbf{E}\left[\sum_{s=1}^{S}\frac{l_s\mathbf{1}[\mathcal{G}_{s,k,i}]}{\sqrt{T_i(t_s)}}\right] \leq 6\bar{c}_i\sum_{s=1}^{S}\left(\sqrt{T_i(t_{s+1})} - \sqrt{T_i(t_s)}\right)\mathbf{1}\left[\sqrt{T_i(t_s)} \leq \gamma\right] \leq 12\bar{c}_i\gamma = 12M\frac{\sqrt{\alpha_k\bar{c}_i\tilde{C}_i\ln T}}{\Delta_i}$ , where we the last inequality follows from Lemma 3.1. Combining this with (25), we obtain  $\mathbf{E}\left[\hat{R}_T\right] = O\left(M\ln T \cdot \sum_{i \in \tilde{A}}\tilde{C}_i\sum_{k=1}^{\infty}\frac{\alpha_k}{\beta_k}\frac{1}{\Delta_i}\right) = O\left(M\sum_{i \in \tilde{A}}\frac{\tilde{C}_i\ln T}{\Delta_i}\right)$ , which completes the proof.
A.6 Proof of the second part of (10) (bounding  $\hat{R}_T^{(2)}$ )
The second part of (10) can be shown via the following:
Lemma A.3. Given  $a_{s}^{\prime}$ ,  $t_s$  and  $t_{s + 1}$ , we have  $\mathbf{E}\left[\sum_{t = t_s}^{t_{s + 1} - 1}(q^\top a_s' - r_t^\top a_t)\right] \leq 3M\frac{C_\mathrm{u}}{C_1}$ .
Proof. For any  $s \geq 1$  and task  $i \in [N]$ , the number of times to start task  $i$  during the  $s$ -th phase is at least  $(T_i(t_{s+1}) - T_i(t_s) - 1)$ . We hence have
$$
\begin{array}{l} \mathbf {E} \left[ \sum_ {t = t _ {s}} ^ {t _ {s + 1} - 1} r _ {t} ^ {\top} a _ {t} \right] \geq \mathbf {E} \left[ \sum_ {i \in A _ {s} ^ {\prime}} \bar {r} _ {i} (T _ {i} (t _ {s + 1}) - T _ {i} (t _ {s}) - 1) \right] \geq \mathbf {E} \left[ \sum_ {i \in A _ {s} ^ {\prime}} \bar {r} _ {i} \frac {l _ {s} - 3 C _ {\mathrm {u}}}{\bar {c} _ {i}} \right] \\ = \mathbf {E} \left[ \sum_ {i \in A _ {s} ^ {\prime}} q _ {i} (l _ {s} - 3 C _ {\mathrm {u}}) \right] \geq \mathbf {E} \left[ l _ {s} \sum_ {i \in A _ {s} ^ {\prime}} q _ {i} \right] - 3 \frac {C _ {\mathrm {u}}}{C _ {\mathrm {l}}} M = \mathbf {E} \left[ l _ {s} \cdot q ^ {\top} a _ {s} ^ {\prime} \right] - 3 \frac {C _ {\mathrm {u}}}{C _ {\mathrm {l}}} M, \\ \end{array}
$$
where the second inequality follows from Lemma A.4 and the third inequality follows from  $q_{i} = \bar{r}_{i} / \bar{c}_{i} \leq 1 / C_{1}$  and  $|A_s'| \leq M$ . This completes the proof.
We used the following lemma in the proof of Lemma A.3.
Lemma A.4. For any  $s \geq 1$  and  $i \in A_s'$ , we have
$$
\mathbf {E} \left[ T _ {i} \left(t _ {s + 1}\right) - T _ {i} \left(t _ {s}\right) \mid l _ {s} \right] \geq \frac {l _ {s} - 2 C _ {\mathrm {u}}}{\bar {c} _ {i}}. \tag {28}
$$
$$
l _ {s} \leq 2 \bar {c} _ {i} \cdot \mathbf {E} _ {s} \left[ T _ {i} \left(t _ {s + 1}\right) - T _ {i} \left(t _ {s}\right) \right], \tag {29}
$$
where  $\mathbf{E}_s$  denote the conditional expectation given  $l_{s}$  and  $T_{i}(t_{s})$ .
Proof. We will show (28) by induction in the value of  $l_{s} \geq 0$ . It is clear that (28) holds if  $l_{s} \leq 2C_{\mathrm{u}}$ . As the inductive hypothesis, we assume that (28) holds if  $l_{s} \leq m$  for some  $m \geq 2C_{\mathrm{u}}$ . Then, if  $l_{s} = m + 1 > 2C_{\mathrm{u}}$ , at least once, task  $i \in A_{s}'$  is started and completed during the  $s$ -th phase. Let  $c_{i}$  denote the processing time when task  $i$  is first started and completed during the  $s$ -th phase. We then have
$$
\begin{array}{l} \mathbf {E} \left[ T _ {i} \left(t _ {s + 1}\right) - T _ {i} \left(t _ {s}\right) \mid l _ {s} = m + 1 \right] = \sum_ {c = C _ {1}} ^ {C _ {\mathrm {u}}} \Pr [ c _ {i} = c ] \mathbf {E} \left[ 1 + T _ {i} \left(t _ {s + 1}\right) - T _ {i} \left(t _ {s}\right) \mid l _ {s} = m + 1 - c \right] \\ = 1 + \sum_ {c = C _ {1}} ^ {C _ {\mathrm {u}}} \operatorname * {P r} [ c _ {i} = c ] \mathbf {E} [ T _ {i} (t _ {s + 1}) - T _ {i} (t _ {s}) | l _ {s} = m + 1 - c ] \\ \geq 1 + \sum_ {c = C _ {1}} ^ {C _ {\mathrm {u}}} \operatorname * {P r} [ c _ {t, i} = c ] \frac {m + 1 - c - 2 C _ {\mathrm {u}}}{\bar {c} _ {i}} \\ = 1 + \frac {m + 1 - \bar {c} _ {i} - 2 C _ {\mathrm {u}}}{\bar {c} _ {i}} = \frac {m + 1 - 2 C _ {\mathrm {u}}}{\bar {c} _ {i}}, \\ \end{array}
$$
where the inequality follows from the inductive hypothesis. Hence, (28) holds for  $l = m + 1$  as well. By induction, it has been shown that (28) holds. Further, as we have  $l_{s} \geq C_{1}B + 2C_{\mathrm{u}} \geq 4C_{\mathrm{u}}$ , we have  $l_{s} - 2C_{\mathrm{u}} \geq l_{s} / 2$ , which implies (29) holds.
From Lemma A.3, we have  $\hat{R}_T^{(2)}\leq 2\frac{C_{\mathrm{u}}}{C_1} SM$ . By combining this with  $S = O\left(\frac{C_{\mathrm{u}}}{C_1} N\ln T\right)$ , which follows from Lemma 3.2, we obtain the second part of (10).
A.7 Proof of Theorem 4.5
We start from the special case in which  $M = 1$ , i.e., we assume  $\mathcal{A} = \{\{i\} \mid i \in [N]\} \cup \{\emptyset\}$ . We will show the following two lower bounds separately:
$$
R _ {T} = \Omega \left(\min  \left\{\sqrt {\frac {1}{C _ {1}} N T}, \frac {T}{C _ {1}} \right\}\right), \tag {30}
$$
$$
R _ {T} = \Omega \left(\frac {C _ {\mathrm {u}} - C _ {\mathrm {l}}}{C _ {\mathrm {l}} C _ {\mathrm {u}}} \min  \left\{\sqrt {C _ {\mathrm {u}} N T}, T \right\} - \frac {C _ {\mathrm {u}}}{C _ {\mathrm {l}}}\right). \tag {31}
$$
These two bounds together lead to the regret lower bound of
$$
R _ {T} = \Omega \left(\frac {1}{C _ {\mathrm {l}}} \min  \left\{\sqrt {C _ {\mathrm {u}} N T}, T \right\}\right). \tag {32}
$$
In fact, if  $C_{\mathrm{u}} \leq 2C_{1}$ , (30) implies that (32) holds as we have  $\frac{C_{\mathrm{u}}}{C_{1}^{2}} = \Theta \left( \frac{1}{C_{1}} \right)$ . On the other hand, if  $C_{\mathrm{u}} \geq 2C_{1}$ , (31) implies that (32) holds since we have  $\frac{C_{\mathrm{u}} - C_{1}}{C_{1}C_{\mathrm{u}}} = \Omega \left( \frac{1}{C_{1}} \right)$ .
From the lower bound of (32), we can obtain
$$
R _ {T} = \Omega \left(\frac {1}{C _ {1}} \min  \left\{\sqrt {C _ {\mathrm {u}} M N T}, M T \right\}\right) \tag {33}
$$