SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we study Tiered MAB and Tiered RL when a single low-tier task  $M_{\mathrm{Lo}}$  is available. The key challenge compared with [13] is that we do not have knowledge about whether  $M_{\mathrm{Lo}}$  and  $M_{\mathrm{Hi}}$  are similar or not so the pure exploitation will not work. Instead, the algorithm should be able to identify whether  $M_{\mathrm{Lo}}$  and  $M_{\mathrm{Hi}}$  are close enough to transfer by data collected so far, and balance between the exploration by itself and the exploitation from  $M_{\mathrm{Lo}}$  at the same time.
To overcome the challenge, we identify a state-wise checking event, such that, under Assump. B, if  $s_h$  is transferable, the event is true almost all the time, and otherwise, every mistake will reduce the uncertainty so the chance the event holds is limited. By utilizing it, our algorithm can wisely switch between optimistic exploration and pessimistic exploitation and achieve robust transfer. In Sec. 4.1, we start with the MAB setting and illustrate the main idea, and in Sec. 4.2, we generalize our techniques to RL setting, and discuss how to overcome the challenges brought by state transition.
4.1 Robust Transfer in Tiered Multi-Armed Bandits
The algorithm is provided in Alg. 1. We choose  $\mathrm{Alg}^{\mathrm{Lo}}$  as UCB, and  $\mathrm{Alg}^{\mathrm{Hi}}$  as an exploitation-or-UCB style algorithm branched by a checking event in line 7, which is the key step to avoid negative transfer.
Algorithm 1: Robust Tiered MAB
1 Initilize:  $\alpha >2$  .  $N_{\mathrm{Lo}}^{1}(i)$ $N_{\mathrm{Hi}}^{1}(i)$ $\widehat{\mu}_{\mathrm{Lo}}^{1}(i)$ $\widehat{\mu}_{\mathrm{Hi}}^{1}(i)\gets 0,\forall i\in \mathcal{A};f(k):= 1 + 16A^{2}(k + 1)^{2}$    
2 Pull each arm at the beginning  $A$  iterations   
3 for  $k = A + 1,A + 2,\ldots ,K$  do   
4  $\overline{\mu}_{\mathrm{Lo}}^k (i)\leftarrow \widehat{\mu}_{\mathrm{Lo}}^k (i) + \sqrt{\frac{2\alpha\log f(k)}{N_{\mathrm{Lo}}^k(i)}}$ $\overline{\pi}_{\mathrm{Lo}}^k\gets \arg \max_i\overline{\mu}_{\mathrm{Lo}}^k (i),\quad \pi_{\mathrm{Lo}}^k\gets \overline{\pi}_{\mathrm{Lo}}^k.$    
5  $\underline{\mu}_{\mathrm{Lo}}^k (i)\leftarrow \widehat{\mu}_{\mathrm{Lo}}^k (i) - \sqrt{\frac{2\alpha\log f(k)}{N_{\mathrm{Lo}}^k(i)}},\quad \underline{\pi}_{\mathrm{Lo}}^k\gets \arg \max_i\underline{\mu}_{\mathrm{Lo}}^k (i).$    
6  $\overline{\mu}_{\mathrm{Hi}}^k (i)\leftarrow \widehat{\mu}_{\mathrm{Hi}}^k (i) + \sqrt{\frac{2\alpha\log f(k)}{N_{\mathrm{Hi}}^k(i)}},\quad \overline{\pi}_{\mathrm{Hi}}^k\gets \arg \max_i\overline{\mu}_{\mathrm{Hi}}^k (i).$    
7 if  $\underline{\mu}_{\mathrm{Lo}}^{k}(\underline{\pi}_{\mathrm{Lo}}^{k})\leq \overline{\mu}_{\mathrm{Hi}}^{k}(\underline{\pi}_{\mathrm{Lo}}^{k}) + \varepsilon$  and  $N_{\mathrm{Lo}}^{k}(\underline{\pi}_{\mathrm{Lo}}^{k}) > k / 2$  then  $\pi_{\mathrm{Hi}}^{k}\gets \underline{\pi}_{\mathrm{Lo}}^{k}$  else  $\pi_{\mathrm{Hi}}^{k}\gets \overline{\pi}_{\mathrm{Hi}}^{k}$    
8 Interact  $M_{\mathrm{Hi}} / M_{\mathrm{Lo}}$  by  $\pi_{\mathrm{Hi}}^{k} / \pi_{\mathrm{Lo}}^{k}$  ; Update  $N_{\mathrm{Lo}}^{k + 1} / N_{\mathrm{Hi}}^{k + 1}$  and empirical mean  $\widehat{\mu}_{\mathrm{Lo}}^{k + 1} / \widehat{\mu}_{\mathrm{Hi}}^{k + 1}$    
9 end
Key Insights: Separation between Transferable and Non-Transferable Cases To understand our checking event, we consider the following two cases: (1)  $M_{\mathrm{Lo}}$  and  $M_{\mathrm{Hi}}$  are  $\varepsilon$ -close, and (2)  $i_{\mathrm{Hi}}^{*} \neq i_{\mathrm{Lo}}^{*}$ .
(in the rest cases we have  $i_{\mathrm{Hi}}^* = i_{\mathrm{Lo}}^*$  but  $\mu_{\mathrm{Lo}}(i_{\mathrm{Lo}}^*) > \mu_{\mathrm{Hi}}(i_{\mathrm{Lo}}^*) + \varepsilon$ , so exploiting from  $M_{\mathrm{Lo}}$  is harmless). Recall Assump. B, in Case 1, we have  $\mu_{\mathrm{Lo}}(i_{\mathrm{Lo}}^*) \leq \mu_{\mathrm{Hi}}(i_{\mathrm{Lo}}^*) + \varepsilon$ , while in Case 2, with an appropriate choice of  $\varepsilon$  (e.g.  $\varepsilon = \frac{\Delta_{\mathrm{min}}}{4} < \frac{\Delta_{\mathrm{min}}}{4}$ ), we have  $\mu_{\mathrm{Lo}}(i_{\mathrm{Lo}}^*) \geq \mu_{\mathrm{Hi}}(i_{\mathrm{Hi}}^*) - \frac{\Delta_{\mathrm{min}}}{2} \geq \mu_{\mathrm{Hi}}(i_{\mathrm{Lo}}^*) + \varepsilon + \frac{\Delta_{\mathrm{Hi}}(i_{\mathrm{Lo}}^*)}{2}$ , which reveals the separation between two cases. As a result, if we can construct an uncertainty-based upper bound  $\overline{\mu}_{\mathrm{Hi}}^k(i_{\mathrm{Lo}}^*)$  for  $\mu_{\mathrm{Hi}}(i_{\mathrm{Lo}}^*)$ , we should expect the event  $\mathcal{E} := \mu_{\mathrm{Lo}}(i_{\mathrm{Lo}}^*) < \overline{\mu}_{\mathrm{Hi}}^k(i_{\mathrm{Lo}}^*) + \varepsilon$  almost always be true in Case 1, while in Case 2, every time  $\mathcal{E}$  occurs and  $\mathrm{Alg}^{\mathrm{Hi}}$  takes  $i_{\mathrm{Lo}}^*$ , the "self-correction" is triggered: the uncertainty is reduced so the estimation  $\overline{\mu}_{\mathrm{Hi}}^k(i_{\mathrm{Lo}}^*)$  gets closer to  $\mu_{\mathrm{Lo}}(i_{\mathrm{Lo}}^*)$ , and because of the separation between  $\mu_{\mathrm{Lo}}(i_{\mathrm{Lo}}^*)$  and  $\mu_{\mathrm{Hi}}(i_{\mathrm{Lo}}^*)$ , the number of times that  $\mathcal{E}$  is true is limited. The remaining issue is that we do not know  $\mu_{\mathrm{Lo}}(i_{\mathrm{Lo}}^*)$  and  $i_{\mathrm{Lo}}^*$ , and we approximate them with LCB value  $\underline{\mu}_{\mathrm{Lo}}^k(\cdot)$  and its greedy policy. The additional checking event  $N_{\mathrm{Lo}}^k(\underline{\pi}_{\mathrm{Lo}}^k) > k/2$  is used to increase the confidence that  $\underline{\pi}_{\mathrm{Lo}}^k = i_{\mathrm{Lo}}^*$  once transfer, which also contributes to reducing the regret. Finally, to achieve constant regret, we use  $\alpha$  to control the total failure rate to be  $\sum_{k=1}^{+\infty} k^{-\Theta(\alpha)} = C$  for some constant  $C$ . We summarize the main result in Thm. 4.1 and defer the proof to Appx. C.
Theorem 4.1. [Tiered MAB with Single Source Tasks] Under Assump. A, B and C, by running Alg. 1 with  $\varepsilon = \frac{\widetilde{\Delta}_{\mathrm{min}}}{4}$  and  $\alpha >2$  , we always have Regret  $K(M_{Hi}) = O(\sum_{\Delta_{Hi}(i) > 0}\frac{1}{\Delta_{Hi}(i)}\log K)$  . Moreover, if  $M_{Hi}$  and  $M_{Lo}$  are  $\frac{\widetilde{\Delta}_{\mathrm{min}}}{4}$  close, we have: Regret  $K(M_{Hi}) = O(\sum_{\Delta_{Hi}(i) > 0}\frac{1}{\Delta_{Hi}(i)}\log \frac{A}{\Delta_{\mathrm{min}}})$
Comparison with [22, 13] As we can see, our algorithm can automatically achieve constant regret if tasks are similar while retaining near-optimal otherwise. Notably, even when  $M_{\mathrm{Hi}} = M_{\mathrm{Lo}}$ , our regret bound  $\widetilde{O}(\sum_{\Delta_{\mathrm{Hi}}(i) > 0} \frac{1}{\Delta_{\mathrm{Hi}}(i)})$  is strictly better than  $\widetilde{O}(\sqrt{\frac{A}{\Delta_{\mathrm{min}}}} \sqrt{\sum_{\Delta_{\mathrm{Hi}}(i) > 0} \frac{1}{\Delta_{\mathrm{Hi}}(i)}})$  in [22] and  $\widetilde{O}(\sum_{\Delta_{\mathrm{Hi}}(i) > 0} (A - i)(\frac{1}{\Delta_{\mathrm{Hi}}(i)} - \frac{\Delta_{\mathrm{Hi}}(i)}{\Delta_{\mathrm{Hi}}(i - 1)^2}))$  in [13] under the same setting.
4.2 Robust Transfer in Tiered Tabular RL
In this section, we focus on RL setting. We provide the algorithm in Alg. 2, where we defer the details of ModelLearning function and the requirements for Bonus function to Appx. D.1. In the following, we first highlight our main result.
Theorem 4.2. [Tiered RL with Single Source Tasks] Under Assump. A, B and C, Cond. D.1 for  $\mathrm{Alg}^{Lo}$  and Cond. D.3 for Bonus function, by running Alg. 2 with  $\varepsilon = \frac{\tilde{\Delta}_{\min}}{4(H + 1)}$ ,  $\alpha > 2$ , an arbitrary  $\lambda > 0$ , we have
$$
\operatorname{Regret}_{K}(M_{Hi}) = O\Big(SH\sum_{h = 1}^{H}\sum_{(s_{h},a_{h})\in \mathcal{S}_{h}\times \mathcal{A}_{h}\setminus \mathcal{C}_{h}^{\lambda}}(\frac{H}{\Delta_{\min}}\wedge \frac{1}{\Delta_{Hi}(s_{h},a_{h})})\log (SAHK)\Big).
$$
Here the set  $\mathcal{C}_h^\lambda \subset S_h\times \mathcal{A}_h$  captures the beneficial state-action pairs to be introduced later. For simplicity, in Thm. 4.2 above, we omit all constant terms independent with  $K$  that may include  $\lambda^{-1}$ ,  $\Delta_{\mathrm{min}}^{-1}$  or  $\log 1 / d_h^* (\cdot)$ . The complete version of Thm. 4.2 can be found in Thm. D.16. As we can see, comparing with pure online learning algorithms [23, 34, 7], in our setting,  $M_{\mathrm{Hi}}$  only suffers non-constant regret on a subset of the state-action space. The  $SH$  factor may be further improved by choosing better Bonus functions than our Example D.4 given in Appx. D.1.
Different from the bandit setting, the state transition causes more challenges. In the following, we first explain the algorithm design to highlight how we overcome the difficulties, and then provide the analysis and proof sketch. Detailed proofs can be found in Appx. D.
Technical Challenges and Algorithm Design Similar to MAB setting, for  $M_{\mathrm{Lo}}$  we choose an arbitrary near-optimal algorithm, and for  $M_{\mathrm{Hi}}$  we set up a state-wise checking event  $\underline{V}_{\mathrm{Lo},h}^{k}(\cdot) \leq \widetilde{Q}_{\mathrm{Hi},h}^{k}(\cdot, \pi_{\mathrm{Lo},h}^{k}) + \varepsilon$  in line 15 to determine whether to exploit from  $M_{\mathrm{Lo}}$  or not. Here  $\underline{V}_{\mathrm{Lo},h}^{k}$  and  $\widetilde{Q}_{\mathrm{Hi},h}^{k}$  serve as lower and upper bounds for  $V_{\mathrm{Lo}}^{*}$  and  $Q_{\mathrm{Hi}}^{*}$ , and  $\underline{V}_{\mathrm{Lo},h}^{k}$  and  $\underline{\pi}_{\mathrm{Lo},h}^{k}$  are constructed by Pessimistic Value Iteration [16], which can be shown to converge to  $V_{\mathrm{Lo},h}^{*}$  and  $\pi_{\mathrm{Lo},h}^{*}$ , respectively. Similar to [13], for the choice of  $\mathrm{Alg}^{\mathrm{Lo}}$  and the bonus term used to construct lower/upper confidence estimation, we consider general algorithm framework under Cond. D.1 and Cond. D.3 in Appx. D.1.
To overcome challenges resulting from state transition, we make two major modifications when moving from MAB to RL setting. First of all, because of the constraint on the optimality of  $\mathrm{Alg}^{\mathrm{L0}}$ ,
Algorithm 2: Robust Tiered RL
1 Input: Ratio  $\lambda \in (0,1)$ ;  $\alpha > 2$ ; Auxiliary functions Bonus and ModelLearning; Sequence of confidence level  $(\delta_k)_{k \geq 1}$  with  $\delta_k = 1/SAHk^\alpha$ ;  $\varepsilon := \widetilde{\Delta}_{\min} / 4(H + 1)$  for some  $\widetilde{\Delta}_{\min} \leq \Delta_{\min}$   
2 Initialize:  $D_{\mathrm{Lo}}^{0} / D_{\mathrm{Hi}}^{0} \gets \{\}$ ;  $\forall k, \underline{V_{(\cdot),H + 1}^k}, \underline{Q_{(\cdot),H + 1}^k}, \widetilde{V}_{\mathrm{Hi},H + 1}^k, \widetilde{Q}_{\mathrm{Hi},H + 1}^k \gets 0$ .  
3 for  $k = 1,2,\ldots$  do  
4  $\pi_{\mathrm{Lo}} \gets \mathrm{Alg}^{\mathrm{Lo}}(D_{\mathrm{Lo}}^{k - 1})$ ;  
5  $\{\widehat{\mathbb{P}}_{\mathrm{Lo},h}^{k}\}_{h = 1}^{H} \gets \text{Model Learning}(D_{\mathrm{Lo}}^{k - 1}), \{b_{\mathrm{Lo},h}^{k}\}_{h = 1}^{H} \gets \text{Bonus}(D_{\mathrm{Lo}}^{k - 1}, \delta_k)$ .  
6 for  $h = H, H - 1,\ldots, 1$  do  
7  $\underline{Q}_{\mathrm{Lo},h}^{k}(\cdot,\cdot) \gets \max \{0, r_{\mathrm{Lo},h}(\cdot,\cdot) + \widehat{\mathbb{P}}_{\mathrm{Lo},h}^{k} \underline{V}_{\mathrm{Lo},h + 1}^{k}(\cdot,\cdot) - b_{\mathrm{Lo},h}^{k}(\cdot,\cdot)\}$ .  
8  $\underline{V}_{\mathrm{Lo},h}^{k}(\cdot) = \max a \underline{Q}_{\mathrm{Lo},h}^{k}(\cdot,a), \quad \underline{\pi}_{\mathrm{Lo},h}^{k}(\cdot) \gets \arg \max a \underline{Q}_{\mathrm{Lo},h}^{k}(\cdot,a)$ .  
9 end  
10  $\{\widehat{\mathbb{P}}_{\mathrm{Hi},h}^{k}\}_{h = 1}^{H} \gets \text{Model Learning}(D_{\mathrm{Hi}}^{k - 1}); \{b_{\mathrm{Hi},h}^{k}\}_{h = 1}^{H} \gets \text{Bonus}(D_{\mathrm{Hi}}^{k - 1}, \delta_k)$ .  
11 for  $h = H, H - 1,\ldots, 1$  do  
12  $\underline{Q}_{\mathrm{Hi},h}^{\pi_{\mathrm{Hi}}}(\cdot,\cdot) \gets \max \{0, r_{\mathrm{Hi},h}(\cdot,\cdot) + \widehat{\mathbb{P}}_{\mathrm{Hi},h}^{k} \underline{V}_{\mathrm{Hi},h + 1}^{k}(\cdot,\cdot) - b_{\mathrm{Hi},h}^{k}(\cdot,\cdot)\}$ .  
13  $\widetilde{Q}_{\mathrm{Hi},h}^{k}(\cdot,\cdot) \gets \min \{H, r_{\mathrm{Hi},h}(\cdot,\cdot) + \widehat{\mathbb{P}}_{\mathrm{Hi},h}^{k} \widetilde{V}_{\mathrm{Hi},h + 1}^{k}(\cdot,\cdot) + b_{\mathrm{Hi},h}^{k}(\cdot,\cdot)\}$ .  
14 for  $s_h \in S_h$  do  
15 if  $V_{Lo,h}^{k}(s_h) \leq \widetilde{Q}_{Hi,h}^{k}(s_h, \underline{\pi}_{Lo,h}^{k}) + \varepsilon$  and  $\max a N_{Lo,h}^{k}(s_h, a) > \frac{\lambda}{3} k$  then  
16 |  $\pi_{\mathrm{Hi}}^{k}(s_h) \gets \arg \max a N_{Lo,h}^{k}(s_h, a)$ . // "Trust and Exploit" Branch  
17 end  
18 else  
19  $\pi_{\mathrm{Hi}}^{k}(\cdot) \gets \arg \max a \widetilde{Q}_{\mathrm{Hi},h}^{k}(\cdot,a)$ . // "Explore by itself" Branch;  
20  
21 |  $V_{Hi,h}^{k}(s_h) \gets \min \{H, Q_{Hi,h}^{k}(s_h, \pi_{Hi}^{k}) + \frac{1}{H}(Q_{Hi,h}^{k}(s_h, \pi_{Hi}^{k}) - Q_{Hi,h}^{\pi_{Hi}}(s_h, \pi_{Hi}^{k}))\}$ .  
22 |  $V_{Hi,h}^{k}(s_h) = Q_{Hi,h}^{\pi_{Hi}}(s_h, \pi_{Hi}^{k})$ .  
23 end  
24 end
we cannot expect  $M_{\mathrm{Lo}}$  to provide useful information on those  $(s_h, a_h)$  with  $d_{\mathrm{Lo}}^*(s_h, a_h) = 0$  since they will not be explored sufficiently. Therefore, in the checking event in line 15, we include  $\max_a N_{\mathrm{Lo},h}^k(s_h, a) > \Theta(\lambda k)$  as a criterion, where  $\lambda$  is a hyper-parameter chosen and input to the algorithm. Intuitively, for all  $s_h, a_h$ , we should expect  $N_{\mathrm{Lo},h}^k(s_h, a_h) \approx \widetilde{O}(d_{\mathrm{Lo}}^*(s_h, a_h) \cdot k)$  when  $k$  is large enough. Therefore, by comparing  $N_{\mathrm{Lo},h}^k$  with  $\lambda k$ , we can filter out those  $s_h$  with  $d_{\mathrm{Lo}}^*(s_h) < \lambda$  to avoid harm from inaccurate estimation.
Secondly and more importantly, different from MAB setting, besides the error occurred at a particular step, we also need to handle the error accumulated during the back-propagation process of value iteration. In our case, this is reflected by the loss of overestimation when we incorporate selective exploitation into the optimism-based exploration framework. To see this, suppose at some  $s_h$ , we have an overestimation on optimal value  $Q_{\mathrm{Hi},h}^*$  denoted as  $\widetilde{Q}_{\mathrm{Hi},h}^k$ . When the checking criterion is satisfied, if we mimic the MAB setting, i.e., assign  $\pi_{\mathrm{Lo},h}^k$  to  $\pi_{\mathrm{Hi},h}^k$  and update value by  $\widetilde{V}_{\mathrm{Hi},h}^k(s_h) \gets \widetilde{Q}_{\mathrm{Hi},h}^k(s_h, \pi_{\mathrm{Hi},h}^k)$ , when  $\pi_{\mathrm{Lo}}^*(s_h) \neq \pi_{\mathrm{Hi},h}^*(s_h)$ ,  $\widetilde{V}_{\mathrm{Hi},h}^k(s_h)$  is no longer guaranteed to be an overestimation for  $V_{\mathrm{Hi},h}^*(s_h)$ . As  $\widetilde{V}_{\mathrm{Hi},h}^k(s_h)$  involves in back-propagation, it will pull down the estimation value for its ancestor states, thus reducing the chance to visit  $s_h$  and slowing down the "self-correction process" which works well in MAB setting.
The key insight to overcome such difficulty is that, if the checking event holds yet  $\pi_{\mathrm{Lo}}^{*}(s_{h}) \neq \pi_{\mathrm{Hi},h}^{*}(s_{h})$ , the gap between  $\widetilde{Q}_{\mathrm{Hi},h}^{k}(s_{h},\pi_{\mathrm{Lo}}^{*})$  and  $Q_{\mathrm{Hi},h}^{*}(s_{h},\pi_{\mathrm{Lo}}^{*})$  should not be small, and we can show that  $\widetilde{Q}_{\mathrm{Hi},h}^{k}(s_{h},\underline{\pi}_{\mathrm{Lo}}^{k}) \approx \widetilde{Q}_{\mathrm{Hi},h}^{k}(s_{h},\pi_{\mathrm{Lo}}^{*}) \geq Q_{\mathrm{Hi},h}^{*}(s_{h},\pi_{\mathrm{Lo}}^{*}) + \Theta (\frac{H}{H + 1}\Delta_{\mathrm{Hi}}(s_{h},\pi_{\mathrm{Lo}}^{*}))$  with the choice of
$\varepsilon = O(\widetilde{\Delta}_{\min} / H)$ . Therefore, revising  $\widetilde{Q}_{\mathrm{Hi},h}^{k}(s_{h},\underline{\pi}_{\mathrm{Lo}}^{k})$  by adding  $1 / H$  of the gap  $\widetilde{Q}_{\mathrm{Hi},h}^{k}(s_{h},\underline{\pi}_{\mathrm{Lo}}^{k}) - Q_{\mathrm{Hi},h}^{*}(s_{h},\pi_{\mathrm{Lo}}^{*})$  (line 19) is enough to guarantee the overestimation. Lastly, since  $Q_{\mathrm{Hi},h}^{*}(s_{h},\pi_{\mathrm{Lo}}^{*})$  in unknown, we construct an underestimation  $\underline{Q}_{\mathrm{Hi},h}^{\pi_{\mathrm{Hi}}^{k}}(s_{h},\pi_{\mathrm{Lo}}^{*})$  and use it instead. As a result, we have the following theorem, where the clip function is defined by  $\operatorname {Clip}[x|w]\coloneqq x\cdot \mathbb{I}[x\geq w]$ .
Theorem 4.3. There exists  $k_{ost} = \text{Poly}(S, A, H, \lambda^{-1}, \Delta_{\min}^{-1})$ , such that, for all  $k \geq k_{ost}$ , on some event  $\mathcal{E}_k$  with  $\mathbb{P}(\mathcal{E}_k) \leq 3\delta_k$ , we have  $Q_{Hi,h}^*(s_h, a_h) \leq \widetilde{Q}_{Hi,h}^k(s_h, a_h)$ ,  $V_{Hi,h}^*(s_h) \leq \widetilde{V}_{Hi,h}^k(s_h)$ ,  $\forall h \in [H]$ ,  $s_h \in S_h$ ,  $a_h \in \mathcal{A}_h$  and
$$
V _ {H i, 1} ^ {*} \left(s _ {1}\right) - V _ {H i, 1} ^ {\pi_ {H i} ^ {k}} \left(s _ {1}\right) \leq 2 e \mathbb {E} _ {\pi_ {H i} ^ {k}} \left[ \sum_ {h = 1} ^ {H} C l i p \left[ \min  \left\{H, 4 b _ {H i, h} ^ {k} \left(s _ {h}, a _ {h}\right) \right\} \right| \frac {\Delta_ {\min }}{4 e H} \vee \frac {\Delta_ {H i} \left(s _ {h} , a _ {h}\right)}{4 e} \right] \Bigg ]. \tag {1}
$$
Benefits of Knowledge Transfer We first take a look at  $k \geq k_{ost}$ . As implied from Eq. (1), we can upper bound the regret on each  $s_h, a_h$  by summing over the RHS of Eq. (1). Note that by Cond. D.3,  $b_{\mathrm{Hi},h}^k(s_h, a_h) = O\left(\frac{\mathrm{Poly}(SAH)\log k}{\sqrt{N_{\mathrm{Hi},h}^k(s_h, a_h)}}\right)$  and  $\mathbb{E}[N_{\mathrm{Hi},h}^k(s_h, a_h)] = \sum_{k'=1}^{k-1} d^{\pi_{\mathrm{Hi}}^k}(s_h, a_h)$ , we can establish the near-optimal regret bound with similar techniques in [23] regardless of the similarity between  $M_{\mathrm{Hi}}$  and  $M_{\mathrm{Lo}}$ . Moreover, because of the knowledge transfer from  $M_{\mathrm{Lo}}$ , we can achieve better regret bounds for  $M_{\mathrm{Hi}}$ . In the following, we characterize three subclasses of state-action pairs, on which  $\mathrm{Alg}^{\mathrm{Hi}}$  only suffers constant regret. First of all, for those  $s_h \in \mathcal{Z}_h^\lambda$ , we can expect the checking event almost always hold for arbitrary  $k$ . Hence, when  $k$  is large enough,  $\pi_{\mathrm{Hi},h}^k(s_h) = \underline{\pi}_{\mathrm{Lo},h}^k(s_h) \approx \pi_{\mathrm{Hi},h}^*(s_h)$ , implying  $\mathrm{Alg}^{\mathrm{Hi}}$  will almost never take sub-optimal actions at  $s_h$  since then. We denote this first subclass as  $\mathcal{C}_h^{1,\lambda} := \{(s_h, a_h) | s_h \in \mathcal{Z}_h^\lambda, a_h \neq \pi_{\mathrm{Hi},h}^*(s_h)\}$ . Secondly, note that, given a state  $s_h$  if all possible trajectories starting from  $s_1$  to  $s_h$  have overlap with  $\mathcal{C}_{h'}^{1,\lambda}$  for some  $h' \in [h-1]$ , when  $k$  is large enough,  $\pi_{\mathrm{Hi}}^k$  will almost have no chance to reach  $s_h$  and will not suffer the regret at  $s_h$ . For convenience, we define function  $\text{Block}(\{\mathcal{C}_{h'}^{1,\lambda}\}_{h'=1}^{h-1}, s_h)$  which takes True for those states described above, and takes False for the others. Then, we define the second subclass by  $\mathcal{C}_h^{2,\lambda} := \{(s_h, a_h) | \text{Block}(\{\mathcal{C}_{h'}^{1,\lambda}\}_{h'=1}^{h-1}, s_h) = \text{True}, s_h \notin \mathcal{Z}_h^\lambda, a_h \in \mathcal{A}_h\}$ . Finally, for those  $s_h, a_h$  with  $d_{\mathrm{Hi}}^*(s_h, a_h) > 0$ , we can show  $N_{\mathrm{Hi},h}^k(s_h, a_h) \approx d_{\mathrm{Hi}}^*(s_h, a_h)k$ . Therefore,  $b_{\mathrm{Hi},h}^k(s_h, a_h) \propto \log k / \sqrt{N_{\mathrm{Hi},h}^k(s_h, a_h)}$  in Eq. (1) will decay and the clipping operator will take effect, which leads to constant regret. This third subclass is denoted by  $\mathcal{C}_h^* := \{(s_h, a_h) | d_{\mathrm{Hi}}^*(s_h, a_h) > 0\}$ . Based on the above discussion, we define  $\mathcal{C}_h^* := \mathcal{C}_h^{\lambda,1} \cup \mathcal{C}_h^{\lambda,2} \cup \mathcal{C}_h^*$  to be the benefitial states set in Thm. 4.2.
For  $k \leq k_{ost}$ , for the lack of overestimation, we simply use  $H$  to upper bound the value gap  $V^{*} - V^{\pi_{\mathrm{Hi}}^{k}}$ . This results in a  $\mathrm{Poly}(S, A, H, \lambda^{-1}, \Delta_{\mathrm{min}}^{-1})$  burn-in term, which was omitted in Thm. 4.2 since it is independent with  $K$ . Besides, by the definition of  $k_{ost}$  in Thm. 4.3, we can see the trade-off of choosing  $\lambda$ : a smaller  $\lambda$  can enlarge  $\mathcal{C}_h^\lambda$  so we have constant regret on more state-action pairs, while it also results in the delay of overestimation by the larger  $k_{ost}$ .
Constant Regret in the Entire MDP We may expect constant regret in the entire  $M_{\mathrm{Hi}}$  in some special cases. Note that, if  $\forall h\in [H],\forall s_h$  with  $d_{\mathrm{Hi}}^* (s_h) > 0$ ,  $s_h\in \mathcal{Z}_h^\lambda$ , we have  $\mathcal{C}_h^\lambda = \mathcal{S}_h\times \mathcal{A}_h$ ,  $\operatorname{Regret}_K(M_{\mathrm{Hi}})$  will be independent w.r.t.  $K$ . From this perspective, if  $\lambda$  is chosen appropriately, e.g.  $\lambda \leq \min_{s_h}d_{\mathrm{Hi}}^* (s_h)$ , we can recover the constant regret under the setting  $M_{\mathrm{Lo}} = M_{\mathrm{Hi}}$  in [13].
Choice of  $\lambda$  In this paper, we do not treat  $\lambda$  as a parameter to optimize. In practice, without prior knowledge about  $\max_{s_h}d_{\mathrm{Lo}}^* (s_h)$ , one may choose  $\lambda = O(1 / S)$  to ensure some chance that transferable states exist, since there exists at least some states satisfying  $d_{\mathrm{Lo}}^* (s_h)\geq 1 / S$ .
5 Robust Tiered MAB/RL with Multiple Low-Tier Tasks
Now, we focus on the case when a source task set  $\mathcal{M}_{\mathrm{Lo}} := \{M_{\mathrm{Lo},w}\}_{w=1}^{W}$  is available (see Frw. 5 in Appx. A). Our objective is to achieve benefits on those states  $s_h$  as long as there exists some task  $w \in [W]$  such that  $M_{\mathrm{Lo},w}$  and  $M_{\mathrm{Hi}}$  are close on  $s_h$ , while retaining near-optimal regret in other cases under Assump. B. The key challenge comparing with single task case is that,  $\mathrm{Alg}^{\mathrm{Hi}}$  has to identify for each state which task in  $\mathcal{M}_{\mathrm{Lo}}$  is the appropriate one to leverage. The main novelty and contribution
in this section is a task selection mechanism we call "Trust till Failure", which can automatically adapt to the similar task if it exists. We first highlight the main results for MAB and RL setting.
Theorem 5.1. [Tiered MAB with Multiple Source Tasks] Under Assump. A, B, and C, by running Alg. 3 with  $\mathcal{M}_{Lo} = \{M_{Lo,w}\}_{w=1}^{W}$  and  $M_{Lo}$ , with  $\varepsilon = \frac{\widetilde{\Delta}_{\min}}{4}$  and  $\alpha > 2$ , we always have:  $\operatorname{Regret}_K(M_{Hi}) = O(\sum_{\Delta_{Hi}(i) > 0} \frac{1}{\Delta_{Hi}(i)} \log(WK))$ . Moreover, if at least one task in  $\mathcal{M}_{Lo}$  is  $\frac{\widetilde{\Delta}_{\min}}{4}$ -close to  $M_{Hi}$ , we further have:  $\operatorname{Regret}_K(M_{Hi}) = O(\sum_{\Delta_{Hi}(i) > 0} \frac{1}{\Delta_{Hi}(i)} \log \frac{AW}{\Delta_{\min}})$ .
Theorem 5.2. [Tiered RL with Multiple Source Tasks] Under Assump. A, B, C, and Cond. D.3, F.4, by running Alg. 7 in Appx. F.1 with  $\varepsilon = \frac{\widetilde{\Delta}_{\min}}{4(H + 1)}$ ,  $\alpha > 2$  and any  $\lambda > 0$ , we have
$$
R e g r e t _ {K} (M _ {H i}) = O \Big (S H \sum_ {h = 1} ^ {H} \sum_ {(s _ {h}, a _ {h}) \in \mathcal {S} _ {h} \times \mathcal {A} _ {h} \backslash \mathcal {C} _ {h} ^ {\lambda , [ W ]}} (\frac {H}{\Delta_ {\operatorname* {m i n}}} \wedge \frac {1}{\Delta_ {H i} (s _ {h} , a _ {h})}) \log (S A H W K) \Big).
$$