SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Robust Knowledge Transfer in Tiered RL
Jiawei Huang
Department of Computer Science ETH Zurich
jiawei.huang@inf.ethz.ch
In this paper, we study the Tiered Reinforcement Learning setting, a parallel transfer learning framework, where the goal is to transfer knowledge from the low-tier (source) task to the high-tier (target) task to reduce the exploration risk of the latter while solving the two tasks in parallel. Unlike previous work, we do not assume the low-tier and high-tier tasks share the same dynamics or reward functions, and focus on robust knowledge transfer without prior knowledge on the task similarity. We identify a natural and necessary condition called the "Optimal Value Dominance" for our objective. Under this condition, we propose novel online learning algorithms such that, for the high-tier task, it can achieve constant regret on partial states depending on the task similarity and retain near-optimal regret when the two tasks are dissimilar, while for the low-tier task, it can keep near-optimal without making sacrifice. Moreover, we further study the setting with multiple low-tier tasks, and propose a novel transfer source selection mechanism, which can ensemble the information from all low-tier tasks and allow provable benefits on a much larger state-action space.
Comparing with individual learning from scratch, transferring knowledge from other similar tasks or side information has been proven to be an effective way to reduce the exploration risk and improve sample efficiency in Reinforcement Learning (RL). Multi-Task RL (MT-RL) [29] and Transfer RL [26, 18, 37] are two mainstream knowledge transfer frameworks; however, both are subject to limitations when dealing with real-world scenarios. MT-RL studies the setting where a set of similar tasks are solved concurrently, and the main objective is to accelerate the learning by sharing information of all tasks together. However, in practice, in many MT-RL scenarios, the tasks are not equally important and we are more interested in the performance of certain tasks. For example, in robot learning, a few robots are more valuable and hard to fix, while the others are cheaper or just simulators. Most existing works on MT-RL treat all tasks equally and focus primarily on the reduction of the total regret of all tasks as a whole [3, 8, 36, 11], with no guarantee of improving a particular task. In contrast, transfer RL distinguishes the priority of different tasks by categorizing them into source and target tasks and aims at transferring the knowledge from source tasks (or some side information like value predictors) to facilitate the learning of target tasks [21, 27, 9, 10]. However, a key assumption in transfer RL is that the source task is completely solved before the learning of the target task, and this is not always practical. For example, in some sim-to-real domain, the source task simulator may require a long time to solve [5], and in some user-interaction scenarios [13], the source and target tasks refer to different user groups and they have to be served simultaneously. In these cases, it's more reasonable to solve the source and target tasks in parallel and transfer the information immediately once available.
Recently, [13] proposed a new "parallel knowledge transfer" framework, called Tiered RL, which is promising to fill the gap. Tiered RL considers the case when a source task  $M_{\mathrm{Lo}}$  and a target task
$M_{\mathrm{Hi}}$  are learned in parallel, by two separate algorithms  $\mathrm{Alg}^{\mathrm{Lo}}$  and  $\mathrm{Alg}^{\mathrm{Hi}}$ , and its goal is to reduce the exploration risk and regret in learning  $M_{\mathrm{Hi}}$  by leveraging knowledge transfer from  $M_{\mathrm{Lo}}$  to  $M_{\mathrm{Hi}}$ . [13] showed that under the strong assumption that  $M_{\mathrm{Lo}} = M_{\mathrm{Hi}}$ , it's possible to achieve constant regret in learning  $M_{\mathrm{Hi}}$  while keeping regret in  $M_{\mathrm{Lo}}$  optimal in gap-dependent setting. Yet, their algorithm based on Pessimistic Value Iteration (PVI) [16] can be hardly applied when the assumption  $M_{\mathrm{Lo}} = M_{\mathrm{Hi}}$  breaks, given its pure exploitation nature, making their result very restrictive.
In this paper, we study the general Tiered RL setting without prior knowledge about how similar the tasks are. The key question we would like to address is: Can we design algorithms s.t.: (1) Regret in  $M_{\mathrm{Lo}}$  keeps near-optimal; (2) Regret in  $M_{\mathrm{Hi}}$  achieves provable benefits when  $M_{\mathrm{Lo}}$  and  $M_{\mathrm{Hi}}$  are similar while retaining near-optimal otherwise? Note for  $\mathrm{Alg}^{\mathrm{Lo}}$ , we expect it to achieve near-optimal regret bounds, which is reasonable since the source task is often important and our results still hold if relaxing it. As for  $\mathrm{Alg}^{\mathrm{Hi}}$ , we expect it to be robust, i.e., it can adaptively exploit from  $M_{\mathrm{Lo}}$  if it is close to  $M_{\mathrm{Hi}}$ , while avoiding negative transfer in other cases. Notably, our setting strictly generalizes [13] and is much more challenging, for balancing the exploitation from  $M_{\mathrm{Lo}}$  and exploration from  $M_{\mathrm{Hi}}$  without prior knowledge of task similarity. We give positive answers to the above question in this paper and demonstrate provable benefits with robust knowledge transfer for Tiered RL framework. Below, we summarize our main contributions in three aspects.
Our first contribution is to identify essential conditions and notions about when and how our objective is achievable. We first provide a mild condition called Optimal Value Dominance (OVD), and in Sec. 3, we justify its necessity to our objective by a lower bound result. Our lower bound holds even if  $M_{\mathrm{Lo}}$  is fully known to the learner, and therefore, it also justifies the necessity of similar assumptions in previous transfer RL literatures [9, 10]. Besides, we introduce the notion of transferable states to characterize states on which  $M_{\mathrm{Hi}}$  is expected to achieve benefits by transferring knowledge from  $M_{\mathrm{Lo}}$ . We believe those findings also provide useful insights for further works.
As our second contribution, in Sec. 4, we propose novel algorithms for Tiered Multi-Armed Bandit (MAB) and Tiered RL, which can achieve robust parallel transfer by balancing between pessimism-based exploitation from  $M_{\mathrm{Lo}}$  and optimism-based online learning in  $M_{\mathrm{Hi}}$ . Depending on the similarity between  $M_{\mathrm{Lo}}$  and  $M_{\mathrm{Hi}}$ , our algorithms can enjoy constant regret on a proportion of state-action pairs or even on the entire  $M_{\mathrm{Hi}}$  by leveraging information from  $M_{\mathrm{Lo}}$ , while timely interrupting negative transfer and retaining near-optimal regret on states dissimilar between two tasks. Moreover, in the bandit setting, our result implies a strictly improved regret bound when  $M_{\mathrm{Hi}} = M_{\mathrm{Lo}}$ , compared with previous results under the same setting [22, 13].
Beyond the single low-tier task setting, in many real-world scenarios, it's reasonable to assume there are multiple different low-tier tasks  $\mathcal{M}_{\mathrm{Lo}} = \{M_{\mathrm{Lo},w}\}_{w=1}^{W}$  available. As our third contribution, in Sec. 5, we extend our algorithm to this setting with a new source task selection mechanism. By novel techniques, we show that, even if each  $M_{\mathrm{Lo},w}$  may only share similarity with  $M_{\mathrm{Hi}}$  on a small portion of states, we are able to ensemble the information from each source task together to achieve constant regret on a much larger state-action space "stitched" from the transferable state-action set in each individual  $M_{\mathrm{Lo},w}$ , at the expense of an additional  $\log W$  factor in regret. Besides, our algorithm is still robust to model difference and retains near-optimal regret in general. Although we only study the Tiered RL setting in this paper, we believe our task selection strategy can be applied to standard transfer RL setting [9, 10] when multiple (partially correct) side information or value predictors are provided, which is an interesting direction for future work. Finally, we conduct experiments in toy examples to verify our theoretical results.
1.1 Closely Related Work
For the lack of space, we only discuss closely related work here and defer the rest to Appx. A.2. The most related to us is the Tiered RL framework [13], which was originally motivated by Tiered structure in user-oriented applications. However, they only studied the case when  $M_{\mathrm{Hi}} = M_{\mathrm{Lo}}$ , which limits the practicability of their algorithms. Although there is a sequence of work studying parallel transfer learning in multi-agent system [25, 19], but they mainly focused on heuristic empirical algorithms and did not have theoretical guarantees.
Transfer RL [37], compared to learning from scratch, can reduces exploration risk of target task by leveraging information in similar source tasks or side information [21, 27, 9, 10]. Comparing with
<sup>1</sup>We defer the detailed framework to Appx. A.1 for completeness, which is the same as Fr. 1 in [13].
transfer RL setting, our parallel transfer setting has some additional challenges. Firstly,  $M_{\mathrm{Hi}}$  can only leverage estimated value/model/optimal policy from  $M_{\mathrm{Lo}}$  with uncertainty, which implies we need additional efforts to control failure events with non-zero probability comparing with normal transfer RL setting. Secondly, the constraints on the optimality of  $\mathrm{Regret}_K(M_{\mathrm{Lo}})$ , although reasonable, restrict the transferable information because in  $M_{\mathrm{Lo}}$  estimation uncertainty can only be controlled on those states frequently visited. Moreover, none of these previous work studies how to leverage multiple partially correct side information like what we did in Sec. 5. In MT-RL setting, the benefits of leveraging information gathered from other tasks has been observed from both empirical and theoretical works [31, 3, 8, 24, 36, 11]. But MTRL treats each task equally, and the reduction of total regret over all tasks does not directly imply benefits achieved in a particular task.
2 Preliminary and Problem Formulation
Tiered Stochastic MAB and Tiered Episodic Tabular RL In Tiered MAB setting, we consider a low-tier task  $M_{\mathrm{Lo}}$  and a high-tier task  $M_{\mathrm{Hi}}$  sharing the arm/action space  $\mathcal{A} = \{1,2,\dots,A\}$ . By pulling the arm  $i\in [A]$  in  $M_{\mathrm{Lo}}$  or  $M_{\mathrm{Hi}}$ , the agent can observe a random variable  $r_{\mathrm{Lo}}(i)$  or  $r_{\mathrm{Hi}}(i)\in [0,1]$ . We will use  $\mu_{\mathrm{Lo}}(i) = \mathbb{E}[r_{\mathrm{Lo}}(i)]$  and  $\mu_{\mathrm{Hi}}(i) = \mathbb{E}[r_{\mathrm{Hi}}(i)]$  to denote the expected return of the  $i$ -th arm in  $M_{\mathrm{Lo}}$  and  $M_{\mathrm{Hi}}$ , respectively, and note that it's possible that  $\mu_{\mathrm{Lo}}(i)\neq \mu_{\mathrm{Hi}}(i)$ .
For Tiered RL, we assume that two tasks  $M_{\mathrm{Lo}} = \{\mathcal{S},\mathcal{A},H,\mathbb{P}_{\mathrm{Lo}},r_{\mathrm{Lo}}\}$  and  $M_{\mathrm{Hi}} = \{\mathcal{S},\mathcal{A},H,\mathbb{P}_{\mathrm{Hi}},r_{\mathrm{Hi}}\}$  share the finite state  $S$  and action space  $\mathcal{A}$  across episode length  $H$  (i.e.  $S_{h} = S, A_{h} = A$  for any  $h \in [H]$ ), but may have different time-dependent transition and reward functions  $\mathbb{P}_{\mathrm{Lo}} = \{\mathbb{P}_{\mathrm{Lo},h}\}_{h=1}^{H}, r_{\mathrm{Lo}} = \{r_{\mathrm{Lo},h}\}_{h=1}^{H}$  and  $\mathbb{P}_{\mathrm{Hi}} = \{\mathbb{P}_{\mathrm{Hi},h}\}_{h=1}^{H}, r_{\mathrm{Hi}} = \{r_{\mathrm{Hi},h}\}_{h=1}^{H}$ . W.l.o.g., we assume the initial state  $s_1$  is fixed, and the reward functions  $r_{\mathrm{Lo}}$  and  $r_{\mathrm{Hi}}$  are deterministic and bounded by [0, 1]. In episodic MDPs, we study the time-dependent policy specified as  $\pi := \{\pi_1, \dots, \pi_H\}$  with  $\pi_h : S_h \to \Delta(\mathcal{A}_h)$  for all  $h \in [H]$ , where  $\Delta(\mathcal{A}_h)$  denotes the probability simplex over the action space. With a slight abuse of notation, when  $\pi_h$  is a deterministic policy, we use  $\pi_h : S_h \to \mathcal{A}_h$  to denote the deterministic mapping. Besides, we use  $Q_h^\pi(s,a) = \mathbb{E}[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) | s_h = s, a_h = a, \pi]$ ,  $V_h^\pi(s) = \mathbb{E}_{a \sim \pi}[Q_h^\pi(s,a)]$  to denote the value function for  $\pi$  at step  $h' \in [H]$ , and denote  $d_h^\pi(\cdot) := \operatorname*{Pr}(s_h = |\pi|)$  and  $d_h^\pi(\cdot, \cdot) := \operatorname*{Pr}(s_h = \cdot, a_h = |\pi|)$  as the state and state-action occupancy w.r.t. policy  $\pi$ . We use  $\pi^*$  to denote the optimal policy, and  $V_h^*, Q_h^*, d_h^*$  as a short note when  $\pi = \pi^*$ . To avoid confusion, we will specify the policy and value functions in  $M_{\mathrm{Lo}}$  and  $M_{\mathrm{Hi}}$  by  $L_{\mathrm{Lo}}$  and  $H_{\mathrm{Hi}}$  in subscription, respectively. For example, in  $M_{\mathrm{Lo}}$  we have  $\pi_{\mathrm{Lo}} := \{\pi_{\mathrm{Lo},1}, \dots, \pi_{\mathrm{Lo},H}\}$ ,  $Q_{\mathrm{Lo},h}^{\pi_{\mathrm{Lo}}} / V_{\mathrm{Lo},h}^{\pi_{\mathrm{Lo}}}$  and  $Q_{\mathrm{Lo},h}^* / V_{\mathrm{Lo},h}^*$ ,  $d_{\mathrm{Lo},h}^{\pi_{\mathrm{Lo}}}, d_{\mathrm{Lo},h}^*$ , and similarly for  $M_{\mathrm{Hi}}$ .
Gap-Dependent Setting Throughout, we focus on gap-dependent setting [17, 23, 34, 7]. Below we introduce the notion of gap for  $M_{\mathrm{Lo}}$  as an example, and those for  $M_{\mathrm{Hi}}$  follows similarly. In MAB case, the gap in  $M_{\mathrm{Lo}}$  w.r.t. arm  $i$  is defined as  $\Delta_{\mathrm{Lo}}(i) \coloneqq \max_{j \in [A]} \mu_{\mathrm{Lo}}(j) - \mu_{\mathrm{Lo}}(i), \forall i \in [A]$ , and for tabular RL setting, we have  $\Delta_{\mathrm{Lo}}(s_h, a_h) \coloneqq V_{\mathrm{Lo}, h}^*(s_h) - Q_{\mathrm{Lo}, h}^*(s_h, a_h), \forall h \in [H], s_h \in S_h, a_h \in A_h$ . We use  $\Delta_{\mathrm{Lo,min}}$  to refer to the minimal gap such that  $\Delta(s_h, a_h) \geq \Delta_{\mathrm{Lo,min}}$  for all non-optimal actions  $a_h$ , and use  $\Delta_{\mathrm{min}} \coloneqq \min \{\Delta_{\mathrm{Lo,min}}, \Delta_{\mathrm{Hi,min}}\}$  to denote the minimal gap over two tasks. In the gap-dependent setting, we assume  $\Delta_{\mathrm{min}} > 0$ .
Knowledge Transfer from Multiple Low-Tier Tasks In this case, we assume there are  $W > 1$  different source tasks  $\mathcal{M}_{\mathrm{Lo}} = \{M_{\mathrm{Lo},w}\}_{w=1}^{W}$  and all the tasks share the same state and action spaces but may have different transition and reward function. We defer the extended framework for this setting to Appx. A. We specify the task index  $w \in [W]$  in sub-scription to distinguish the notation for different source tasks (e.g.  $\mathbb{P}_{\mathrm{Lo},w,h}, Q_{\mathrm{Lo},w,h}^{(\cdot)}$ ). Moreover, we define  $\Delta_{\mathrm{min}} := \min \{\Delta_{\mathrm{Lo},1,\min}, \dots, \Delta_{\mathrm{Lo},W,\min}, \Delta_{\mathrm{Hi,min}}\}$ . For convenience, in the rest of the paper, we use TRL-MST (Tiered RL with Multiple Source Task) as a short abbreviation for this setting.
Performance Measure We use Pseudo-Regret as performance measure:  $\mathrm{Regret}_K(M_{\mathrm{Lo}}) := \mathbb{E}\left[\sum_{k=1}^{K} V_1^*(s_1) - V_1^{\pi_{\mathrm{Lo}}^k}(s_1)\right]$ ;  $\mathrm{Regret}_K(M_{\mathrm{Hi}}) := \mathbb{E}\left[\sum_{k=1}^{K} V_1^*(s_1) - V_1^{\pi_{\mathrm{Hi}}^k}(s_1)\right]$ , where  $K$  is the number of iterations,  $\{\pi_{\mathrm{Lo}}^k\}_{k=1}^K$  and  $\{\pi_{\mathrm{Hi}}^k\}_{k=1}^K$  are generated by the algorithms.
Frequently Used Notations We denote  $[n] = \{1,2,\dots,n\}$ . Given a transition matrix  $\mathbb{P}:\mathcal{S}\times \mathcal{A}\to \Delta (\mathcal{S})$ , and a function  $V:S\rightarrow \mathbb{R}$ , we use  $\mathbb{P}V(s_h,a_h)$  as a short note of  $\mathbb{E}_{s^{\prime}\sim \mathbb{P}(\cdot |\cdot ,a)}[V(s^{\prime})]$ . We will use  $i_{\mathrm{Lo}}^{*} / i_{\mathrm{Hi}}^{*}$  to denote the optimal arm in bandit setting, and  $\pi_{\mathrm{Lo}}^{*} / \pi_{\mathrm{Hi}}^{*}$  denotes the optimal policy in RL setting. In TRL-MST setting, we use  $i_{\mathrm{Lo},w}^{*} / \pi_{\mathrm{Lo},w}^{*}$  to distinguish different source tasks.
2.1 Assumptions and Characterization of Transferable States
Throughout the paper, we make several assumptions. The first one is the uniqueness of the optimal policy, which is common in the literature [22, 4].
Assumption A. Both  $M_{\mathrm{Lo}}$  (or  $\{M_{\mathrm{Lo},w}\}_{w = 1}^{W}$ ) and  $M_{\mathrm{Hi}}$  have unique optimal arms/policies.
Next, we introduce a new concept called "Optimal Value Dominance" (OVD for short), which says that for each state (or at least those states reachable by optimal policy in  $M_{\mathrm{Lo}}$ ), the optimal value of  $M_{\mathrm{Lo}}$  is an approximate overestimation for the optimal value of  $M_{\mathrm{Hi}}$ . In Sec. 3, we will use a lower bound to show such a condition is necessary to attain the robust transfer objective.
Assumption B. In single source task setting, we assume  $M_{\mathrm{Lo}}$  has Optimal Value Dominance (OVD) over  $M_{\mathrm{Hi}}$ , s.t.,  $\forall h \in [H]$ , for all  $s_h \in S_h$  (or only for those  $s_h$  with  $d_{\mathrm{Lo},h}^*(s_h) > 0$ ), we have:  $V_{\mathrm{Lo},h}^*(s_h) \geq V_{\mathrm{Hi},h}^*(s_h) - \frac{\Delta_{\min}}{2(H + 1)}$ . In TRL-MST setting, we assume each  $M_{\mathrm{Lo},w}$  has OVD over  $M_{\mathrm{Hi}}$ .<sup>2</sup>
We remark that Assump. B is a rather mild condition that naturally holds with reward shaping. Note that since  $V_{\mathrm{Hi},h}^{*}(\cdot)\leq H - h$ , by shifting the reward function of  $M_{\mathrm{Lo}}$  to  $r_{\mathrm{Lo},h}^{\prime}(\cdot ,\cdot) = r_{\mathrm{Lo},h}(\cdot ,\cdot) + 1$  we immediately obtain the OVD property. Even though, such a reward shift may impair the set of transferable states as we will introduce in Def. 2.2. We provide several reasonable settings in Appx. A.3 including identical model [13], small model difference, and known model difference, where Assump. B is satisfied and there exists a non-empty set of transferable states. We also point out that several existing work on transfer RL assumed something similar or even stronger [9, 10], which we defer a thorough discussion to Appx. A.2.
Assumption C. The learner has access to a quantity  $\widetilde{\Delta}_{\mathrm{min}}$  satisfying  $0 \leq \widetilde{\Delta}_{\mathrm{min}} \leq \Delta_{\mathrm{min}}$ .
The final one is about the knowledge of a lower bound of  $\Delta_{\mathrm{min}}$ , which can always be satisfied by choosing  $\widetilde{\Delta}_{\mathrm{min}} = 0$ . Nevertheless, it would be more beneficial if the learner has access to some quantity  $\widetilde{\Delta}_{\mathrm{min}}$  closer to  $\Delta_{\mathrm{min}}$  than 0. As we introduce below, the magnitude of  $\widetilde{\Delta}_{\mathrm{min}}$  is related to how we quantify the similarity between  $M_{\mathrm{Lo}}$  and  $M_{\mathrm{Hi}}$  and which states we expect to benefit from knowledge transfer. Below we focus on the single source task setting and defer the discussion for TRL-MST setting to Sec. 5.
Definition 2.1 ( $\varepsilon$ -Close). Task  $M_{\mathrm{Hi}}$  is  $\varepsilon$ -close to task  $M_{\mathrm{Lo}}$  on  $s_h$  at step  $h$  for some  $\varepsilon > 0$ , if  $V_{\mathrm{Lo},h}^{*}(s_h) - V_{\mathrm{Hi},h}^{*}(s_h) \leq \varepsilon$  and  $\pi_{\mathrm{Hi}}^{*}(s_h) = \pi_{\mathrm{Lo}}^{*}(s_h)$ .
Definition 2.2 ( $\lambda$ -Transferable States). State  $s_h$  is  $\lambda$ -transferable for some  $\lambda > 0$ , if  $d_{\mathrm{Lo}}^*(s_h) > \lambda$  and  $M_{\mathrm{Hi}}$  is  $\frac{\widetilde{\Delta}_{\mathrm{min}}}{4(H + 1)}$ -close to  $M_{\mathrm{Lo}}$  on  $s_h$ . The set of  $\lambda$ -transferable states at  $h \in [H]$  is denoted as  $\mathcal{Z}_h^\lambda$ .
We regard  $s_h$  in  $M_{\mathrm{Lo}}$  has transferable knowledge to  $M_{\mathrm{Hi}}$ , if it can be reached by optimal policy in  $M_{\mathrm{Lo}}$  and the optimal value and action at  $s_h$  for two tasks are similar. Here the condition  $d_{\mathrm{Lo}}^*(s_h) > 0$  is necessary since in  $M_{\mathrm{Lo}}$ , only the states reachable by  $\pi_{\mathrm{Lo}}^*$  can be explored sufficiently by  $\mathrm{Alg}^{\mathrm{Lo}}$  due to its near optimal regret. Combining with Assump. B, one can observe that the value difference on transferable states are controlled by  $|V_{\mathrm{Lo},h}^*(s_h) - V_{\mathrm{Hi},h}^*(s_h)| = O(\frac{\tilde{\Delta}_{\min}}{H}) \leq O(\frac{\Delta_{\min}}{H})$ . As we will show in Thm. 3.2 in Sec. 3, the term  $O(\Delta_{\min})$  is indeed unimprovable if we expect robustness.
3 Lower Bound Results: Necessary Condition for Robust Transfer
Now we establish lower bounds that show Assump. B is necessary and how the magnitude of  $\Delta_{\mathrm{min}}$  restricts the robust transfer objective. The results in this section are based on two-armed Bernoulli bandits for simplicity, and the proofs are deferred to Appx. B. By extending these hard instances to RL case, there is a gap caused by the additional  $\frac{1}{H}$  in Assump. B and Def. 2.2, which comes from the requirement of our algorithm design, and we leave it to the future work.
Justification for Assump. B We show that if Assump. B is violated, it is impossible to have algorithms  $(\mathrm{Alg}^{\mathrm{Lo}},\mathrm{Alg}^{\mathrm{Hi}})$  to simultaneously achieve constant regret when  $M_{\mathrm{Lo}} = M_{\mathrm{Hi}}$ , while retaining sub-linear regret for all the cases regardless of the similarity between  $M_{\mathrm{Lo}}$  and  $M_{\mathrm{Hi}}$ . Here we require constant regret on  $M_{\mathrm{Lo}} = M_{\mathrm{Hi}}$  since we believe it is a minimal expectation to achieve benefits in transfer when
For convenience, we include the bandit setting as a special case with  $H = 0$ ; see also Def. 2.1, 2.2 and F.1.
the two tasks are identical. Intuitively, without Assump. B, even if we know  $\mu_{\mathrm{Lo}}(i_{\mathrm{Lo}}^{*}) = \mu_{\mathrm{Hi}}(i_{\mathrm{Lo}}^{*})$  , we cannot ensure  $i_{\mathrm{Lo}}^*$  is the optimal arm in  $M_{\mathrm{Hi}}$  . Then, if  $(\mathrm{Alg}^{\mathrm{Lo}},\mathrm{Alg}^{\mathrm{Hi}})$  can achieve constant regret on  $M_{\mathrm{Lo}} = M_{\mathrm{Hi}}$  , the algorithm must stop exploration on the arm  $i\neq i_{\mathrm{Lo}}^{*}$  after finite steps, and thus, it suffers from linear regret in another instance of  $M_{\mathrm{Hi}}$  where  $i_{\mathrm{Lo}}^{*}\neq i_{\mathrm{Hi}}^{*}$
Moreover, Thm. 3.1 holds even if the learner has full information of  $M_{\mathrm{Lo}}$ , where the setting degenerates to normal transfer RL since there is no need to explore  $M_{\mathrm{Lo}}$ . This explains why similar assumptions to Assump. B are considered in previous transfer RL works [9, 10].
Theorem 3.1. Under the violation of Assump.  $B$ , even regardless of the optimality of  $\mathrm{Alg}^{Lo}$ , for each algorithm pair  $(\mathrm{Alg}^{Lo}, \mathrm{Alg}^{Hi})$ , it cannot simultaneously (1) achieve constant regret for the case when  $M_{Lo} = M_{Hi}$  and (2) ensure sub-linear regret in all the other cases.
$\Delta_{\mathrm{min}}$  in Tolerance Error is Inevitable Next, we show that, if  $M_{\mathrm{Hi}}$  and  $M_{\mathrm{Lo}}$  are  $\Delta$ -close for some  $\Delta \geq \frac{\Delta_{\mathrm{min}}}{2}$ , in general we cannot expect to achieve constant regret on  $M_{\mathrm{Hi}}$  by leveraging  $M_{\mathrm{Lo}}$  without other loss. Similar to Thm. 3.1, the main idea is to construct different instances for  $M_{\mathrm{Hi}}$  with different optimal arms and cannot be distinguished within finite number of trials.
Theorem 3.2. [Transferable States are Restricted by  $\Delta_{\mathrm{min}}$ ] Under Assump.  $B$ , regardless of the optimality of  $Alg^{Lo}$ , given arbitrary  $\Delta_{\mathrm{min}}$  and arbitrary  $\Delta \in [\frac{\Delta_{\mathrm{min}}}{2}, \Delta_{\mathrm{min}}]$ , for each algorithm pair  $(Alg^{Lo}, Alg^{Hi})$ , it cannot simultaneously (1) achieve constant regret for the case when  $M_{Lo}$  and  $M_{Hi}$  with minimal gap  $\Delta_{\mathrm{min}}$  are  $\Delta$ -close, and (2) ensure sub-linear regret in all other cases.
4 Robust Tiered MAB/RL with Single Source Task