Title: Robust Knowledge Transfer in Tiered Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning Theory

Transfer RL

Tiered RL
Abstract: In this paper, we study the Tiered Reinforcement Learning setting, a parallel transfer learning framework, where the goal is to transfer knowledge from the low-tier (source) task to the high-tier (target) task to reduce the exploration risk of the latter while solving the two tasks in parallel. Unlike previous work, we do not assume the low-tier and high-tier tasks share the same dynamics or reward functions, and focus on robust knowledge transfer without prior knowledge on the task similarity. We identify a natural and necessary condition called the ``Optimal Value Dominance'' for our objective. Under this condition, we propose novel online learning algorithms such that, for the high-tier task, it can achieve constant regret on partial states depending on the task similarity and retain near-optimal regret when the two tasks are dissimilar, while for the low-tier task, it can keep near-optimal without making sacrifice. Moreover, we further study the setting with multiple low-tier tasks, and propose a novel transfer source selection mechanism, which can ensemble the information from all low-tier tasks and allow provable benefits on a much larger state-action space.

Robust Knowledge Transfer in Tiered RL

Abstract

In this paper, we study the Tiered Reinforcement Learning setting, a parallel transfer learning framework, where the goal is to transfer knowledge from the low-tier (source) task to the high-tier (target) task to reduce the exploration risk of the latter while solving the two tasks in parallel. Unlike previous work, we do not assume the low-tier and high-tier tasks share the same dynamics or reward functions, and focus on robust knowledge transfer without prior knowledge on the task similarity. We identify a natural and necessary condition called the 'Optimal Value Dominance' for our objective. Under this condition, we propose novel online learning algorithms such that, for the high-tier task, it can achieve constant regret on partial states depending on the task similarity and retain near-optimal regret when the two tasks are dissimilar, while for the low-tier task, it can keep near-optimal without making sacrifice. Moreover, we further study the setting with multiple low-tier tasks, and propose a novel transfer source selection mechanism, which can ensemble the information from all low-tier tasks and allow provable benefits on a much larger state-action space.

1 Introduction

Comparing with individual learning from scratch, transferring knowledge from other similar tasks or side information has been proven to be an effective way to reduce the exploration risk and improve sample efficiency in Reinforcement Learning (RL). Multi-Task RL (MT-RL) [29] and Transfer RL [26, 18, 37] are two mainstream knowledge transfer frameworks; however, both are subject to limitations when dealing with real-world scenarios. MT-RL studies the setting where a set of similar tasks are solved concurrently, and the main objective is to accelerate the learning by sharing information of all tasks together. However, in practice, in many MT-RL scenarios, the tasks are not equally important and we are more interested in the performance of certain tasks. For example, in robot learning, a few robots are more valuable and hard to fix, while the others are cheaper or just simulators. Most existing works on MT-RL treat all tasks equally and focus primarily on the reduction of the total regret of all tasks as a whole [3, 8, 36, 11], with no guarantee of improving a particular task. In contrast, transfer RL distinguishes the priority of different tasks by categorizing them into source and target tasks and aims at transferring the knowledge from source tasks (or some side information like value predictors) to facilitate the learning of target tasks [21, 27, 9, 10]. However, a key assumption in transfer RL is that the source task is completely solved before the learning of the target task, and this is not always practical. For example, in some sim-to-real domain, the source task simulator may require a long time to solve [5], and in some user-interaction scenarios [13], the source and target tasks refer to different user groups and they have to be served simultaneously. In these cases, it's more reasonable to solve the source and target tasks in parallel and transfer the information immediately once available.

Recently, [13] proposed a new 'parallel knowledge transfer' framework, called Tiered RL, which is promising to fill the gap. Tiered RL considers the case when a source task M Lo and a target task

M Hi are learned in parallel, by two separate algorithms Alg Lo and Alg Hi , and its goal is to reduce the exploration risk and regret in learning M Hi by leveraging knowledge transfer from M Lo to M Hi. [13] showed that under the strong assumption that M Lo = M Hi, it's possible to achieve constant regret in learning M Hi while keeping regret in M Lo optimal in gap-dependent setting. Yet, their algorithm based on Pessimistic Value Iteration (PVI) [16] can be hardly applied when the assumption M Lo = M Hi breaks, given its pure exploitation nature, making their result very restrictive.

In this paper, we study the general Tiered RL setting without prior knowledge about how similar the tasks are 1 . The key question we would like to address is: Can we design algorithms s.t.: (1) Regret in M Lo keeps near-optimal; (2) Regret in M Hi achieves provable benefits when M Lo and M Hi are similar while retaining near-optimal otherwise? Note for Alg Lo , we expect it to achieve near-optimal regret bounds, which is reasonable since the source task is often important and our results still hold if relaxing it. As for Alg Hi , we expect it to be robust , i.e., it can adaptively exploit from M Lo if it is close to M Hi, while avoiding negative transfer in other cases. Notably, our setting strictly generalizes [13] and is much more challenging, for balancing the exploitation from M Lo and exploration from M Hi without prior knowledge of task similarity. We give positive answers to the above question in this paper and demonstrate provable benefits with robust knowledge transfer for Tiered RL framework. Below, we summarize our main contributions in three aspects.

Our first contribution is to identify essential conditions and notions about when and how our objective is achievable. We first provide a mild condition called Optimal Value Dominance (OVD), and in Sec. 3, we justify its necessity to our objective by a lower bound result. Our lower bound holds even if M Lo is fully known to the learner, and therefore, it also justifies the necessity of similar assumptions in previous transfer RL literatures [9, 10]. Besides, we introduce the notion of transferable states to characterize states on which M Hi is expected to achieve benefits by transferring knowledge from M Lo. We believe those findings also provide useful insights for further works.

As our second contribution , in Sec. 4, we propose novel algorithms for Tiered Multi-Armed Bandit (MAB) and Tiered RL, which can achieve robust parallel transfer by balancing between pessimismbased exploitation from M Lo and optimism-based online learning in M Hi. Depending on the similarity between M Lo and M Hi, our algorithms can enjoy constant regret on a proportion of state-action pairs or even on the entire M Hi by leveraging information from M Lo, while timely interrupting negative transfer and retaining near-optimal regret on states dissimilar between two tasks. Moreover, in the bandit setting, our result implies a strictly improved regret bound when M Hi = M Lo, compared with previous results under the same setting [22, 13].

Beyond the single low-tier task setting, in many real-world scenarios, it's reasonable to assume there are multiple different low-tier tasks M Lo = { M Lo ,w } W w =1 available. As our third contribution , in Sec. 5, we extend our algorithm to this setting with a new source task selection mechanism. By novel techniques, we show that, even if each M Lo ,w may only share similarity with M Hi on a small portion of states, we are able to ensemble the information from each source task together to achieve constant regret on a much larger state-action space 'stitched' from the transferable state-action set in each individual M Lo ,w , at the expense of an additional log W factor in regret. Besides, our algorithm is still robust to model difference and retains near-optimal regret in general. Although we only study the Tiered RL setting in this paper, we believe our task selection strategy can be applied to standard transfer RL setting [9, 10] when multiple (partially correct) side information or value predictors are provided, which is an interesting direction for future work. Finally, we conduct experiments in toy examples to verify our theoretical results.

1.1 Closely Related Work

For the lack of space, we only discuss closely related work here and defer the rest to Appx. A.2. The most related to us is the Tiered RL framework [13], which was originally motivated by Tiered structure in user-oriented applications. However, they only studied the case when M Hi = M Lo, which limits the practicability of their algorithms. Although there is a sequence of work studying parallel transfer learning in multi-agent system [25, 19], but they mainly focused on heuristic empirical algorithms and did not have theoretical guarantees.

Transfer RL [37], compared to learning from scratch, can reduces exploration risk of target task by leveraging information in similar source tasks or side information [21, 27, 9, 10]. Comparing with

transfer RL setting, our parallel transfer setting has some additional challenges. Firstly, M Hi can only leverage estimated value/model/optimal policy from M Lo with uncertainty, which implies we need additional efforts to control failure events with non-zero probability comparing with normal transfer RL setting. Secondly, the constraints on the optimality of Regret K ( M Lo ) , although reasonable, restrict the transferable information because in M Lo estimation uncertainty can only be controlled on those states frequently visited. Moreover, none of these previous work studies how to leverage multiple partially correct side information like what we did in Sec. 5. In MT-RL setting, the benefits of leveraging information gathered from other tasks has been observed from both empirical and theoretical works [31, 3, 8, 24, 36, 11]. But MTRL treats each task equally, and the reduction of total regret over all tasks does not directly imply benefits achieved in a particular task.

2 Preliminary and Problem Formulation

Tiered Stochastic MAB and Tiered Episodic Tabular RL In Tiered MAB setting, we consider a low-tier task M Lo and a high-tier task M Hi sharing the arm/action space A = { 1 , 2 , ..., A } . By pulling the arm i ∈ [ A ] in M Lo or M Hi, the agent can observe a random variable r Lo ( i ) or r Hi ( i ) ∈ [0 , 1] . We will use µ Lo ( i ) = E [ r Lo ( i )] and µ Hi ( i ) = E [ r Hi ( i )] to denote the expected return of the i -th arm in M Lo and M Hi, respectively, and note that it's possible that µ Lo ( i ) = µ Hi ( i ) .

̸

For Tiered RL, we assume that two tasks M Lo = {S , A , H, P Lo , r Lo } and M Hi = {S , A , H, P Hi , r Hi } share the finite state S and action space A across episode length H (i.e. S h = S , A h = A for any h ∈ [ H ] ), but may have different time-dependent transition and reward functions P Lo = { P Lo ,h } H h =1 , r Lo = { r Lo ,h } H h =1 and P Hi = { P Hi ,h } H h =1 , r Hi = { r Hi ,h } H h =1 . W.l.o.g., we assume the initial state s 1 is fixed, and the reward functions r Lo and r Hi are deterministic and bounded by [0 , 1] . In episodic MDPs, we study the time-dependent policy specified as π := { π 1 , ..., π H } with π h : S h → ∆( A h ) for all h ∈ [ H ] , where ∆( A h ) denotes the probability simplex over the action space. With a slight abuse of notation, when π h is a deterministic policy, we use π h : S h →A h to denote the deterministic mapping. Besides, we use Q π h ( s, a ) = E [ ∑ H h ′ = h r h ′ ( s h ′ , a h ′ ) | s h = s, a h = a, π ] , V π h ( s ) = E a ∼ π [ Q π h ( s, a )] to denote the value function for π at step h ∈ [ H ] , and denote d π h ( · ) := Pr( s h = ·| π ) and d π h ( · , · ) := Pr( s h = · , a h = ·| π ) as the state and state-action occupancy w.r.t. policy π . We use π ∗ to denote the optimal policy, and V ∗ h , Q ∗ h , d ∗ h as a short note when π = π ∗ . To avoid confusion, we will specify the policy and value functions in M Lo and M Hi by Lo and Hi in subscription, respectively. For example, in M Lo we have π Lo := { π Lo , 1 , ..., π Lo ,H } , Q π Lo Lo ,h /V π Lo Lo ,h and Q ∗ Lo ,h /V ∗ Lo ,h , d π Lo Lo ,h , d ∗ Lo ,h , and similarly for M Hi.

Gap-Dependent Setting Throughout, we focus on gap-dependent setting [17, 23, 34, 7]. Below we introduce the notion of gap for M Lo as an example, and those for M Hi follows similarly. In MAB case, the gap in M Lo w.r.t. arm i is defined as ∆ Lo ( i ) := max j ∈ [ A ] µ Lo ( j ) -µ Lo ( i ) , ∀ i ∈ [ A ] , and for tabular RL setting, we have ∆ Lo ( s h , a h ) := V ∗ Lo ,h ( s h ) -Q ∗ Lo ,h ( s h , a h ) , ∀ h ∈ [ H ] , s h ∈ S h , a h ∈ A h . We use ∆ Lo , min to refer to the minimal gap such that ∆( s h , a h ) ≥ ∆ Lo , min for all non-optimal actions a h , and use ∆ min := min { ∆ Lo , min , ∆ Hi , min } to denote the minimal gap over two tasks. In the gap-dependent setting, we assume ∆ min > 0 .

Knowledge Transfer from Multiple Low-Tier Tasks In this case, we assume there are W > 1 different source tasks M Lo = { M Lo ,w } W w =1 and all the tasks share the same state and action spaces but may have different transition and reward function. We defer the extended framework for this setting to Appx. A. We specify the task index w ∈ [ W ] in sub-scription to distinguish the notation for different source tasks (e.g. P Lo ,w,h , Q ( · ) Lo ,w,h ). Moreover, we define ∆ min := min { ∆ Lo , 1 , min , ..., ∆ Lo ,W, min , ∆ Hi , min } . For convenience, in the rest of the paper, we use TRL-MST (Tiered RL with Multiple Source Task) as a short abbreviation for this setting.

Performance Measure We use Pseudo-Regret as performance measure: Regret K ( M Lo ) := E [ ∑ K k =1 V ∗ 1 ( s 1 ) -V π k Lo 1 ( s 1 ) ] ; Regret K ( M Hi ) := E [ ∑ K k =1 V ∗ 1 ( s 1 ) -V π k Hi 1 ( s 1 ) ] , where K is the number of iterations, { π k Lo } K k =1 and { π k Hi } K k =1 are generated by the algorithms.

Frequently Used Notations We denote [ n ] = { 1 , 2 , ..., n } . Given a transition matrix P : S × A → ∆( S ) , and a function V : S → R , we use P V ( s h , a h ) as a short note of E s ′ ∼ P ( ·| s,a ) [ V ( s ′ )] . We will use i ∗ Lo /i ∗ Hi to denote the optimal arm in bandit setting, and π ∗ Lo /π ∗ Hi denotes the optimal policy in RL setting. In TRL-MST setting, we use i ∗ Lo ,w /π ∗ Lo ,w to distinguish different source tasks.

2.1 Assumptions and Characterization of Transferable States

Throughout the paper, we make several assumptions. The first one is the uniqueness of the optimal policy, which is common in the literature [22, 4].

Assumption A. Both M Lo (or { M Lo ,w } W w =1 ) and M Hi have unique optimal arms/policies.

Next, we introduce a new concept called 'Optimal Value Dominance' (OVD for short), which says that for each state (or at least those states reachable by optimal policy in M Lo), the optimal value of M Lo is an approximate overestimation for the optimal value of M Hi. In Sec. 3, we will use a lower bound to show such a condition is necessary to attain the robust transfer objective.

Assumption B. In single source task setting, we assume M Lo has Optimal Value Dominance (OVD) over M Hi, s.t., ∀ h ∈ [ H ] , for all s h ∈ S h (or only for those s h with d ∗ Lo ,h ( s h ) > 0 ), we have: V ∗ Lo ,h ( s h ) ≥ V ∗ Hi ,h ( s h ) -∆ min 2( H +1) . In TRL-MST setting, we assume each M Lo ,w has OVD over M Hi. 2

We remark that Assump. B is a rather mild condition that naturally holds with reward shaping. Note that since V ∗ Hi ,h ( · ) ≤ H -h , by shifting the reward function of M Lo to r ′ Lo ,h ( · , · ) = r Lo ,h ( · , · ) + 1 , we immediately obtain the OVD property. Even though, such a reward shift may impair the set of transferable states as we will introduce in Def. 2.2. We provide several reasonable settings in Appx. A.3 including identical model [13], small model difference, and known model difference, where Assump. B is satisfied and there exists a non-empty set of transferable states. We also point out that several existing work on transfer RL assumed something similar or even stronger [9, 10], which we defer a thorough discussion to Appx. A.2.

Assumption C. The learner has access to a quantity ˜ ∆ min satisfying 0 ≤ ˜ ∆ min ≤ ∆ min .

The final one is about the knowledge of a lower bound of ∆ min , which can always be satisfied by choosing ˜ ∆ min = 0 . Nevertheless, it would be more beneficial if the learner has access to some quantity ˜ ∆ min closer to ∆ min than 0. As we introduce below, the magnitude of ˜ ∆ min is related to how we quantify the similarity between M Lo and M Hi and which states we expect to benefit from knowledge transfer. Below we focus on the single source task setting and defer the discussion for TRL-MST setting to Sec. 5.

Definition 2.1 ( ε -Close) . Task M Hi is ε -close to task M Lo on s h at step h for some ε > 0 , if V ∗ Lo ,h ( s h ) -V ∗ Hi ,h ( s h ) ≤ ε and π ∗ Hi ( s h ) = π ∗ Lo ( s h ) .

Definition 2.2 ( λ -Transferable States) . State s h is λ -transferable for some λ > 0 , if d ∗ Lo ( s h ) > λ and M Hi is ˜ ∆ min 4( H +1) -close to M Lo on s h . The set of λ -transferable states at h ∈ [ H ] is denoted as Z λ h .

We regard s h in M Lo has transferable knowledge to M Hi, if it can be reached by optimal policy in M Lo and the optimal value and action at s h for two tasks are similar. Here the condition d ∗ Lo ( s h ) > 0 is necessary since in M Lo, only the states reachable by π ∗ Lo can be explored sufficiently by Alg Lo due to its near optimal regret. Combining with Assump. B, one can observe that the value difference on transferable states are controlled by | V ∗ Lo ,h ( s h ) -V ∗ Hi ,h ( s h ) | = O ( ˜ ∆ min H ) ≤ O ( ∆ min H ) . As we will show in Thm. 3.2 in Sec. 3, the term O (∆ min ) is indeed unimprovable if we expect robustness.

3 Lower Bound Results: Necessary Condition for Robust Transfer

Now we establish lower bounds that show Assump. B is necessary and how the magnitude of ∆ min restricts the robust transfer objective. The results in this section are based on two-armed Bernoulli bandits for simplicity, and the proofs are deferred to Appx. B. By extending these hard instances to RL case, there is a gap caused by the additional 1 H in Assump. B and Def. 2.2, which comes from the requirement of our algorithm design, and we leave it to the future work.

Justification for Assump. B Weshowthat if Assump. B is violated, it is impossible to have algorithms ( Alg Lo , Alg Hi ) to simultaneously achieve constant regret when M Lo = M Hi, while retaining sub-linear regret for all the cases regardless of the similarity between M Lo and M Hi. Here we require constant regret on M Lo = M Hi since we believe it is a minimal expectation to achieve benefits in transfer when

̸

the two tasks are identical. Intuitively, without Assump. B, even if we know µ Lo ( i ∗ Lo ) = µ Hi ( i ∗ Lo ) , we cannot ensure i ∗ Lo is the optimal arm in M Hi. Then, if ( Alg Lo , Alg Hi ) can achieve constant regret on M Lo = M Hi, the algorithm must stop exploration on the arm i = i ∗ Lo after finite steps, and thus, it suffers from linear regret in another instance of M Hi where i ∗ Lo = i ∗ Hi .

Moreover, Thm. 3.1 holds even if the learner has full information of M Lo, where the setting degenerates to normal transfer RL since there is no need to explore M Lo. This explains why similar assumptions to Assump. B are considered in previous transfer RL works [9, 10].

Theorem 3.1. Under the violation of Assump. B, even regardless of the optimality of Alg Lo , for each algorithm pair ( Alg Lo , Alg Hi ) , it cannot simultaneously (1) achieve constant regret for the case when M Lo = M Hi and (2) ensure sub-linear regret in all the other cases.

∆ min in Tolerance Error is Inevitable Next, we show that, if M Hi and M Lo are ∆ -close for some ∆ ≥ ∆ min 2 , in general we cannot expect to achieve constant regret on M Hi by leveraging M Lo without other loss. Similar to Thm. 3.1, the main idea is to construct different instances for M Hi with different optimal arms and cannot be distinguished within finite number of trials.

Theorem 3.2. [Transferable States are Restricted by ∆ min ] Under Assump. B, regardless of the optimality of Alg Lo , given arbitrary ∆ min and arbitrary ∆ ∈ [ ∆ min 2 , ∆ min ] , for each algorithm pair ( Alg Lo , Alg Hi ) , it cannot simultaneously (1) achieve constant regret for the case when M Lo and M Hi with minimal gap ∆ min are ∆ -close, and (2) ensure sub-linear regret in all other cases.

4 Robust Tiered MAB/RL with Single Source Task

In this section, we study Tiered MAB and Tiered RL when a single low-tier task M Lo is available. The key challenge compared with [13] is that we do not have knowledge about whether M Lo and M Hi are similar or not so the pure exploitation will not work. Instead, the algorithm should be able to identify whether M Lo and M Hi are close enough to transfer by data collected so far, and balance between the exploration by itself and the exploitation from M Lo at the same time.

To overcome the challenge, we identify a state-wise checking event, such that, under Assump. B, if s h is transferable, the event is true almost all the time, and otherwise, every mistake will reduce the uncertainty so the chance the event holds is limited. By utilizing it, our algorithm can wisely switch between optimistic exploration and pessimistic exploitation and achieve robust transfer. In Sec. 4.1, we start with the MAB setting and illustrate the main idea, and in Sec. 4.2, we generalize our techniques to RL setting, and discuss how to overcome the challenges brought by state transition.

4.1 Robust Transfer in Tiered Multi-Armed Bandits

The algorithm is provided in Alg. 1. We choose Alg Lo as UCB, and Alg Hi as an exploitation-or-UCB style algorithm branched by a checking event in line 7, which is the key step to avoid negative transfer.

Algorithm 1: Robust Tiered MAB

̸

Key Insights: Separation between Transferable and Non-Transferable Cases To understand our checking event, we consider the following two cases: (1) M Lo and M Hi are ε -close, and (2) i ∗ Hi = i ∗ Lo

̸

(in the rest cases we have i ∗ Hi = i ∗ Lo but µ Lo ( i ∗ Lo ) > µ Hi ( i ∗ Lo ) + ε , so exploiting from M Lo is harmless). Recall Assump. B, in Case 1, we have µ Lo ( i ∗ Lo ) ≤ µ Hi ( i ∗ Lo ) + ε , while in Case 2, with an appropriate choice of ε (e.g. ε = ˜ ∆ min 4 < ∆ min 4 ), we have µ Lo ( i ∗ Lo ) ≥ µ Hi ( i ∗ Hi ) -∆ min 2 ≥ µ Hi ( i ∗ Lo ) + ε + ∆ Hi ( i ∗ Lo ) 2 , which reveals the separation between two cases. As a result, if we can construct an uncertainty-based upper bound µ k Hi ( i ∗ Lo ) for µ Hi ( i ∗ Lo ) , we should expect the event E := µ Lo ( i ∗ Lo ) < µ k Hi ( i ∗ Lo ) + ε almost always be true in Case 1, while in Case 2, everytime E occurs and Alg Hi takes i ∗ Lo , the 'self-correction' is triggered: the uncertainty is reduced so the estimation µ k Hi ( i ∗ Lo ) gets closer to µ Lo ( i ∗ Lo ) , and because of the separation between µ Lo ( i ∗ Lo ) and µ Hi ( i ∗ Lo ) , the number of times that E is true is limited. The remaining issue is that we do not know µ Lo ( i ∗ Lo ) and i ∗ Lo , and we approximate them with LCB value µ k Lo ( · ) and its greedy policy. The additional checking event N k Lo ( π k Lo ) > k/ 2 is used to increase the confidence that π k Lo = i ∗ Lo once transfer, which also contributes to reducing the regret. Finally, to achieve constant regret, we use α to control the total failure rate to be ∑ + ∞ k =1 k -Θ( α ) = C for some constant C . We summarize the main result in Thm. 4.1 and defer the proof to Appx. C.

Theorem 4.1. [Tiered MAB with Single Source Tasks] Under Assump. A, B and C, by running Alg. 1 with ε = ˜ ∆ min 4 and α > 2 , we always have Regret K ( M Hi ) = O ( ∑ ∆ Hi ( i ) > 0 1 ∆ Hi ( i ) log K ) . Moreover, if M Hi and M Lo are ˜ ∆ min 4 -close, we have: Regret K ( M Hi ) = O ( ∑ ∆ Hi ( i ) > 0 1 ∆ Hi ( i ) log A ∆ min ) .

Comparison with [22, 13] As we can see, our algorithm can automatically achieve constant regret if tasks are similar while retaining near-optimal otherwise. Notably, even when M Hi = M Lo, our regret bound ˜ O ( ∑ ∆ Hi ( i ) > 0 1 ∆ Hi ( i ) ) is strictly better than ˜ O ( √ A ∆ min √∑ ∆ Hi ( i ) > 0 1 ∆ Hi ( i ) ) in [22] and ˜ O ( ∑ ∆ Hi ( i ) > 0 ( A -i )( 1 ∆ Hi ( i ) -∆ Hi ( i ) ∆ Hi ( i -1) 2 )) in [13] under the same setting.

4.2 Robust Transfer in Tiered Tabular RL

In this section, we focus on RL setting. We provide the algorithm in Alg. 2, where we defer the details of ModelLearning function and the requirements for Bonus function to Appx. D.1. In the following, we first highlight our main result.

Theorem 4.2. [Tiered RL with Single Source Tasks] Under Assump. A, B and C, Cond. D.1 for Alg Lo and Cond. D.3 for Bonus function, by running Alg. 2 with ε = ˜ ∆ min 4( H +1) , α > 2 , an arbitrary λ > 0 , we have

Here the set C λ h ⊂ S h ×A h captures the benefitable state-action pairs to be introduced later. For simplicity, in Thm. 4.2 above, we omit all constant terms independent with K that may include λ -1 , ∆ -1 min or log 1 /d ∗ h ( · ) . The complete version of Thm. 4.2 can be found in Thm. D.16. As we can see, comparing with pure online learning algorithms [23, 34, 7], in our setting, M Hi only suffers non-constant regret on a subset of the state-action space. The SH factor may be further improved by choosing better Bonus functions than our Example D.4 given in Appx. D.1.

Different from the bandit setting, the state transition causes more challenges. In the following, we first explain the algorithm design to highlight how we overcome the difficulties, and then provide the analysis and proof sketch. Detailed proofs can be found in Appx. D.

Technical Challenges and Algorithm Design Similar to MAB setting, for M Lo we choose an arbitrary near-optimal algorithm, and for M Hi we set up a state-wise checking event V k Lo ,h ( · ) ≤ ˜ Q k Hi ,h ( · , π k Lo ,h ) + ε in line 15 to determine whether to exploit from M Lo or not. Here V k Lo ,h and ˜ Q k Hi ,h serve as lower and upper bounds for V ∗ Lo and Q ∗ Hi , and V k Lo ,h and π k Lo ,h are constructed by Pessimistic Value Iteration [16], which can be shown to converge to V ∗ Lo ,h and π ∗ Lo ,h , respectively. Similar to [13], for the choice of Alg Lo and the bonus term used to construct lower/upper confidence estimation, we consider general algorithm framework under Cond. D.1 and Cond. D.3 in Appx. D.1.

To overcome challenges resulting from state transition, we make two major modifications when moving from MAB to RL setting. First of all, because of the constraint on the optimality of Alg Lo ,

Algorithm 2: Robust Tiered RL

we cannot expect M Lo to provide useful information on those ( s h , a h ) with d ∗ Lo ( s h , a h ) = 0 since they will not be explored sufficiently. Therefore, in the checking event in line 15, we include max a N k Lo ,h ( s h , a ) > Θ( λk ) as a criterion, where λ is a hyper-parameter chosen and input to the algorithm. Intuitively, for all s h , a h , we should expect N k Lo ,h ( s h , a h ) ≈ ˜ O ( d ∗ Lo ( s h , a h ) · k ) when k is large enough. Therefore, by comparing N k Lo ,h with λk , we can filter out those s h with d ∗ Lo ( s h ) < λ to avoid harm from inaccurate estimation.

̸

Secondly and more importantly, different from MAB setting, besides the error occurred at a particular step, we also need to handle the error accumulated during the back-propagation process of value iteration. In our case, this is reflected by the loss of overestimation when we incorporate selective exploitation into the optimism-based exploration framework. To see this, suppose at some s h , we have an overestimation on optimal value Q ∗ Hi ,h denoted as ˜ Q k Hi ,h . When the checking criterion is satisfied, if we mimic the MAB setting, i.e., assign π k Lo ,h to π k Hi ,h and update value by ˜ V k Hi ,h ( s h ) ← ˜ Q k Hi ,h ( s h , π k Hi ,h ) , when π ∗ Lo ( s h ) = π ∗ Hi ,h ( s h ) , ˜ V k Hi ,h ( s h ) is no longer guaranteed to be an overestimation for V ∗ Hi ,h ( s h ) . As ˜ V k Hi ,h ( s h ) involves in back-propagation, it will pull down the estimation value for its ancestor states, thus reducing the chance to visit s h and slowing down the 'self-correction process' which works well in MAB setting.

̸

The key insight to overcome such difficulty is that, if the checking event holds yet π ∗ Lo ( s h ) = π ∗ Hi ,h ( s h ) , the gap between ˜ Q k Hi ,h ( s h , π ∗ Lo ) and Q ∗ Hi ,h ( s h , π ∗ Lo ) should not be small, and we can show that ˜ Q k Hi ,h ( s h , π k Lo ) ≈ ˜ Q k Hi ,h ( s h , π ∗ Lo ) ≥ Q ∗ Hi ,h ( s h , π ∗ Lo ) + Θ( H H +1 ∆ Hi ( s h , π ∗ Lo )) with the choice of

ε = O ( ˜ ∆ min /H ) . Therefore, revising ˜ Q k Hi ,h ( s h , π k Lo ) by adding 1 /H of the gap ˜ Q k Hi ,h ( s h , π k Lo ) -Q ∗ Hi ,h ( s h , π ∗ Lo ) (line 19) is enough to guarantee the overestimation. Lastly, since Q ∗ Hi ,h ( s h , π ∗ Lo ) in unknown, we construct an underestimation Q π k Hi Hi ,h ( s h , π ∗ Lo ) and use it instead. As a result, we have the following theorem, where the clip function is defined by Clip [ x | w ] := x · I [ x ≥ w ] .

Theorem 4.3. There exists k ost = Poly ( S, A, H, λ -1 , ∆ -1 min ) , such that, for all k ≥ k ost , on some event E k with P ( E k ) ≤ 3 δ k , we have Q ∗ Hi ,h ( s h , a h ) ≤ ˜ Q k Hi ,h ( s h , a h ) , V ∗ Hi ,h ( s h ) ≤ ˜ V k Hi ,h ( s h ) , ∀ h ∈ [ H ] , s h ∈ S h , a h ∈ A h and

Benefits of Knowledge Transfer We first take a look at k ≥ k ost . As implied from Eq. (1), we can upper bound the regret on each s h , a h by summing over the RHS of Eq. (1). Note that by Cond. D.3, b k Hi ,h ( s h , a h ) = O ( Poly ( SAH ) log k √ N k Hi ,h ( s h ,a h ) ) and E [ N k Hi ,h ( s h , a h )] = ∑ k -1 k ′ =1 d π k Hi ( s h , a h ) , we can establish the

̸

near-optimal regret bound with similar techniques in [23] regardless of the similarity between M Hi and M Lo. Moreover, because of the knowledge transfer from M Lo, we can achieve better regret bounds for M Hi. In the following, we characterize three subclasses of state-action pairs, on which Alg Hi only suffers constant regret. First of all , for those s h ∈ Z λ h , we can expect the checking event almost always hold for arbitrary k . Hence, when k is large enough, π k Hi ,h ( s h ) = π k Lo ,h ( s h ) ≈ π ∗ Hi ,h ( s h ) , implying Alg Hi will almost never take sub-optimal actions at s h since then. We denote this first subclass as C 1 ,λ h := { ( s h , a h ) | s h ∈ Z λ h , a h = π ∗ Hi ,h ( s h ) } . Secondly , note that, given a state s h , if all possible trajectories starting from s 1 to s h have overlap with C 1 ,λ h ′ for some h ′ ∈ [ h -1] , when k is large enough, π k Hi will almost have no chance to reach s h and will not suffer the regret at s h . For convenience, we define function Block ( {C 1 ,λ h ′ } h -1 h ′ =1 , s h ) which takes True for those states described above, and takes False for the others. Then, we define the second subclass by C 2 ,λ h := { ( s h , a h ) | Block ( {C 1 ,λ h ′ } h -1 h ′ =1 , s h ) = True , s h ̸∈ Z λ h , a h ∈ A h } . Finally , for those s h , a h with d ∗ Hi ( s h , a h ) > 0 , we can show N k Hi ,h ( s h , a h ) ≈ d ∗ Hi ( s h , a h ) k . Therefore, b k Hi ,h ( s h , a h ) ∝ log k/ √ N k Hi ,h ( s h , a h ) in Eq. (1) will decay and the clipping operator will take effect, which leads to constant regret. This third subclass is denoted by C ∗ h := { ( s h , a h ) | d ∗ Hi ( s h , a h ) > 0 } . Based on the above discussion, we define C λ h := C λ, 1 h ∪ C λ, 2 h ∪ C ∗ h to be the benefitable states set in Thm. 4.2.

For k ≤ k ost , for the lack of overestimation, we simply use H to upper bound the value gap V ∗ -V π k Hi . This results in a Poly ( S, A, H, λ -1 , ∆ -1 min ) burn-in term, which was omitted in Thm. 4.2 since it is independent with K . Besides, by the definition of k ost in Thm. 4.3, we can see the trade-off of choosing λ : a smaller λ can enlarge C λ h so we have constant regret on more state-action pairs, while it also results in the delay of overestimation by the larger k ost .

Constant Regret in the Entire MDP We may expect constant regret in the entire M Hi in some special cases. Note that, if ∀ h ∈ [ H ] , ∀ s h with d ∗ Hi ( s h ) > 0 , s h ∈ Z λ h , we have C λ h = S h × A h , Regret K ( M Hi ) will be independent w.r.t. K . From this perspective, if λ is chosen appropriately, e.g. λ ≤ min s h d ∗ Hi ( s h ) , we can recover the constant regret under the setting M Lo = M Hi in [13].

Choice of λ In this paper, we do not treat λ as a parameter to optimize. In practice, without prior knowledge about max s h d ∗ Lo ( s h ) , one may choose λ = O (1 /S ) to ensure some chance that transferable states exist, since there exists at least some states satisfying d ∗ Lo ( s h ) ≥ 1 /S .

5 Robust Tiered MAB/RL with Multiple Low-Tier Tasks

Now, we focus on the case when a source task set M Lo := { M Lo ,w } W w =1 is available (see Frw. 5 in Appx. A). Our objective is to achieve benefits on those states s h as long as there exists some task w ∈ [ W ] such that M Lo ,w and M Hi are close on s h , while retaining near-optimal regret in other cases under Assump. B. The key challenge comparing with single task case is that, Alg Hi has to identify for each state which task in M Lo is the appropriate one to leverage. The main novelty and contribution

in this section is a task selection mechanism we call 'Trust till Failure' , which can automatically adapt to the similar task if it exists. We first highlight the main results for MAB and RL setting.

Theorem 5.1. [Tiered MAB with Multiple Source Tasks] Under Assump. A, B, and C, by running Alg. 3 with M Lo = { M Lo ,w } W w =1 and M Lo, with ε = ˜ ∆ min 4 and α > 2 , we always have: Regret K ( M Hi ) = O ( ∑ ∆ Hi ( i ) > 0 1 ∆ Hi ( i ) log( WK )) . Moreover, if at least one task in M Lo is ˜ ∆ min 4 -close to M Hi, we further have: Regret K ( M Hi ) = O ( ∑ ∆ Hi ( i ) > 0 1 ∆ Hi ( i ) log AW ∆ min ) .

Theorem 5.2. [Tiered RL with Multiple Source Tasks] Under Assump. A, B, C, and Cond. D.3, F.4, by running Alg. 7 in Appx. F .1 with ε = ˜ ∆ min 4( H +1) , α > 2 and any λ > 0 , we have

Algorithm 3: Robust Tiered MAB with Multiple Source Tasks

̸

̸

For the lack of space, in the following, we only analyze the bandit setting to explain the key idea of our task selection strategy. For the RL setting, we defer to Appx. F the algorithm Alg. 7, detailed version of Thm. 5.2 (Thm. D.16), defintion of transferable set C λ, [ W ] h (Def. F.2), and technical details.

Algorithm Design and Proof Sketch for Bandit Setting The algorithm is provided in Alg. 3. Comparing with Alg. 1 in single task setting, the main difference is the task selection strategy from line 9 to line 17. We first examine each source task with a checking event similar to single task setting, and collect those feasible tasks passing the test to I k . Intuitively, for those M Lo ,w ∗ close to M Lo, we expect M Lo ,w ∈ I k holds almostly for arbitrary k > 0 , while for the other M Lo ,w ′ , if it takes the position of w k , following M Lo ,w ′ will reduce the uncertainty and it will be ruled out from I k , eventually. So we expect w k can 'escape' from dissimilar source tasks but be absorbed to the similar task if exists. Therefore, if I k is empty, Alg Hi will do exploration by itself. Otherwise, we choose one from I k to transfer the action until it fails on the checking event. However, for any ε the algorithm chosen, those 'marginally similar' source tasks (denoted as M Lo , ˜ w ), which are ε ′ -close to M Hi for some ε ′ only slightly larger than ε , may cause some trouble. Because the checking event

will finally eliminate M Lo , ˜ w since they are not ε -close, but it may occupy the position w k for a long time before elimination, especially when ε ′ is extremely close to ε . After eliminating M Lo , ˜ w , Alg Hi needs to re-select one from I k . Now since other sub-optimal arms in M Hi haven't been chosen for a long time and the confidence level δ k = O (1 /k α ) is decreasing, I k will include those dissimilar tasks again, which causes difficulty to identify the true similar task. To solve this issue, once the previous trusted task fails, we give priority to the task recommending the same action as the previous one (line 14). As a result, since M Lo , ˜ w and M Lo ,w ∗ share the optimal action, after the elimination of M Lo , ˜ w , we can expect w k to only switch among those tasks M Lo ,w with π ∗ Lo ,w = π ∗ Hi . We highlight this technical novelty to Lem. 5.3 below, and defer all the proofs to Appx. E.

̸

Lemma 5.3. [Absorbing to Similar Task] Under Assump. A, B and C, there exists a constant c ∗ , s.t., if there exists at least one w ∗ ∈ [ W ] such that M Lo ,w ∗ is ˜ ∆ min 4 -close to M Hi, by running Alg. 3 with ε = ˜ ∆ min 4 and α > 2 , for any k ≥ k ∗ := c ∗ αA ∆ 2 min log αAW ∆ min , we have Pr( π k Hi = i ∗ Hi ) = O ( A k 2 α -2 ) .

6 Experiments

In this section, we evaluate our most representative algorithm, Alg. 7, in multiple source tasks setting.

Experiments Setting 3 We set S = A = 3 and H = 5 . The details for construction of source and target tasks are defered to Appx. G. We adapt StrongEuler in [23] as online learning algorithm to solve source tasks, and use the bonus function in [23] as the bonus function in our Alg. 7. We evaluate our algorithm when W = 0 , 1 , 2 , 5 , where W = 0 means the high-tier task M Hi is simply solved by normal online learning method (StrongEuler) without any parallel knowledge transfer. We choose λ = 0 . 3 ≈ 1 /S in Alg. 7, and in the MDP instance we test, across all S · H = 15 states, for W = 1 , 2 , 5 , the number of transferable states would be 6, 9 and 13, respectively.

We choose iteration number K = 1 e 7 , where we start the transfer since k = 5 e 5 to avoid large 'burn-in' terms. As we can see, after the transfer starts, the regret in target task will suddenly increase for a while, because the target task has to make some mistakes and learn from it as a result of the model uncertainty. However, because of our algorithm design, the negative transfer will terminate after a very short period. As predicted by our theory, by adding more and more source tasks which can introduce new transferable states, the target task will suffer less and less regret.

0.5

(transfer starts)

2

4

6

8

10

Iterations (1e6)

5600

5700

5800

5900

6000

6100

Regret

W = 0

W = 1

W = 2

W = 5

7 Conclusion and Future Work

In this paper, we study how to do robust parallel transfer RL when single or multiple source tasks are avilable, without knowledge on models' similarity. The possible future directions include relaxing assumptions, better strategies to leveraging multiple source tasks, and identifying mild structural assumptions allowing for more aggressive transfer, and we defer to Appx. A.4 for more details.

Acknowledgments and Disclosure of Funding

The authors would like to thank Andreas Krause for valuable discussion. The work is supported by ETH research grant and Swiss National Science Foundation (SNSF) Project Funding No. 200021207343.

References

Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep , pages 10-4, 2019.

Mohammad Gheshlaghi Azar, Ian Osband, and R´ emi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

Emma Brunskill and Lihong Li. Sample complexity of multi-task reinforcement learning, 2013.

Jinglin Chen and Nan Jiang. Offline reinforcement learning under value and density-ratio realizability: the power of gaps. arXiv preprint arXiv:2203.13935 , 2022.

HeeSun Choi, Cindy Crump, Christian Duriez, Asher Elmquist, Gregory Hager, David Han, Frank Hearl, Jessica Hodgins, Abhinandan Jain, Frederick Leve, et al. On the use of simulation in robotics: Opportunities, challenges, and suggestions for moving forward. Proceedings of the National Academy of Sciences , 118(1):e1907856118, 2021.

Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning, 2017.

Christoph Dann, Teodor Vanislavov Marinov, Mehryar Mohri, and Julian Zimmert. Beyond value-function gaps: Improved instance-dependent regret bounds for episodic reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/ forum?id=CU8qQMhB3dh .

Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, Jan Peters, et al. Sharing knowledge in multi-task deep reinforcement learning. In 8th International Conference on Learning Representations, { ICLR } 2020, Addis Ababa, Ethiopia, April 26-30, 2020 , pages 1-11. OpenReview. net, 2020.

Noah Golowich and Ankur Moitra. Can q-learning be improved with advice? In Conference on Learning Theory , pages 4548-4619. PMLR, 2022.

Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham M. Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview. net/forum?id=D-X3kH-BkpN .

Jiachen Hu, Xiaoyu Chen, Chi Jin, Lihong Li, and Liwei Wang. Near-optimal representation learning for linear bandits and linear rl. In International Conference on Machine Learning , pages 4349-4358. PMLR, 2021.

Jiawei Huang, Jinglin Chen, Li Zhao, Tao Qin, Nan Jiang, and Tie-Yan Liu. Towards deploymentefficient reinforcement learning: Lower bound and optimality. arXiv preprint arXiv:2202.06450 , 2022.

Jiawei Huang, Li Zhao, Tao Qin, Wei Chen, Nan Jiang, and Tie-Yan Liu. Tiered reinforcement learning: Pessimism in the face of uncertainty and constant regret. arXiv preprint arXiv:2205.12418 , 2022.

Nan Jiang and Jiawei Huang. Minimax value interval for off-policy evaluation and policy optimization. Advances in Neural Information Processing Systems , 33:2747-2758, 2020.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? arXiv preprint arXiv:1807.03765 , 2018.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Tor Lattimore and Csaba Szepesv´ ari. Bandit algorithms . Cambridge University Press, 2020.

Alessandro Lazaric. Transfer in reinforcement learning: a framework and a survey. In Reinforcement Learning , pages 143-173. Springer, 2012.

Yongyuan Liang and Bangwei Li. Parallel knowledge transfer in multi-agent reinforcement learning, 2020.

Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. Advances in neural information processing systems , 33:1264-1274, 2020.

Timothy A Mann and Yoonsuck Choe. Directed exploration in reinforcement learning with transferred knowledge. In European Workshop on Reinforcement Learning , pages 59-76. PMLR, 2013.

Chlo´ e Rouyer and Yevgeny Seldin. Tsallis-inf for decoupled exploration and exploitation in multi-armed bandits. In Conference on Learning Theory , pages 3227-3249. PMLR, 2020.

Max Simchowitz and Kevin G Jamieson. Non-asymptotic gap-dependent regret bounds for tabular mdps. Advances in Neural Information Processing Systems , 32, 2019.

Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with contextbased representations. In International Conference on Machine Learning , pages 9767-9779. PMLR, 2021.

Adam Taylor, Ivana Dusparic, Maxime Gu´ eriau, and Siobh´ an Clarke. Parallel transfer learning in multi-agent systems: What, when and how to transfer? In 2019 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2019.

Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research , 10(7), 2009.

Volodymyr Tkachuk, Sriram Ganapathi Subramanian, and Matthew E. Taylor. The effect of q-function reuse on the total regret of tabular, model-free, reinforcement learning, 2021.

Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226 , 2021.

Nelson Vithayathil Varghese and Qusay H Mahmoud. A survey of multi-task deep reinforcement learning. Electronics , 9(9):1363, 2020.

Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data in online reinforcement learning. arXiv preprint arXiv:2211.04974 , 2022.

Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning , pages 1015-1022, 2007.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellmanconsistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:6683-6694, 2021.

Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems , 34:27395-27407, 2021.

Haike Xu, Tengyu Ma, and Simon Du. Fine-grained gap-dependent bounds for tabular mdps via adaptive multi-step bootstrap. In Conference on Learning Theory , pages 4438-4472. PMLR, 2021.

Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730-2775. PMLR, 2022.

Chicheng Zhang and Zhi Wang. Provably efficient multi-task reinforcement learning with model transfer. Advances in Neural Information Processing Systems , 34:19771-19783, 2021.

Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprint arXiv:2009.07888 , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an extension of \"Tiered-RL\", a multi-fidelity RL framework where a \"low-fidelity\" environment is executed in parallel with the \"high-fidelity\" environment, with the purpose of training faster while keeping near-optimal regret. The paper is a theoretical exploration without empirical evaluation\n\nSTRENGTHS:\n- The paper presents a deep theoretical evaluation of a setting that is very relevant yet not very methodologically-explored: When we have related tasks of varied importances to be solved (such as sim2real), and one task can be leveraged to learn faster another one.\n- An algorithm able to guarantee near-optimal regret while training in multiple tasks might be of use in security-critical applications such as robotics or medical domains.\n\nWEAKNESSES:\n- While the theoretical results sound exciting, I would expect at least a simple empirical evaluation of the proposed algorithm to be provided to show how hard it is to actually implement the algorithm in a practical domain. \n\n- The \"Tiered RL\" setting sounds very similar (if not exactly the same) as the multi-fidelity RL modeling, that wasn't even cited by the authors. While there is still a novelty in the exact way the problem is solved by running all \"tiers\" (or fidelities) in parallel, the problem formulation seems exactly like multi-fidelity MDPs to me, and I would suggest to use the same multi-fidelity MDP formulation to keep consistency in the literature. Multi-fidelity MDPs are explicitly modeled in this paper:\n \nSilva, Felipe Leno, et al. \"Toward Multi-Fidelity Reinforcement Learning for Symbolic Optimization.\", Adaptive and Learning Agents (ALA) workshop, 2023.\n\nAnd the multi-fidelity RL problem has been explored in a similar way in the following papers\n\nSami Khairy and Prasanna Balaprakash. 2022. Multifidelity reinforcement learning with control variates. arXiv preprint arXiv:2206.05165 (2022).\n\nMark Cutler, Thomas J Walsh, and Jonathan P How. 2014. Reinforcement learning with multi-fidelity simulators. In 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 3888–3895.\n\nAt the very least all of those papers should have been included to your related works section.\n\nQUESTIONS:\n- A doubt regarding the setting. Would the agent care about the performance in all tiers? Or is it only important to optimize the performance in the highest tier? If only the highest tier matter, the setting is exactly the same as in multi-fidelity RL as said in the \"weaknesses\" section, otherwise, I would need more clarification of in which practical application this modeling would be useful.\n\nLIMITATIONS:\nNo foreseeable negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-10"}

comments: SUMMARY:
The authors look at the tiered reinforcement learning setting, which is a parallel transfer learning framework where the goal is to transfer knowledge from a low-tier source task to a high-tier target task in order to reduce the exploration risk of the high-tier task. Additionally, these tasks are solved in parallel. Contrary to previous related work, the authors do not assume the low-tier and high-tier tasks share dynamics or reward functions and focus on robust knowledge transfer without prior knowledge on task similarity. The authors use a condition called the “Optimal Value Dominance” to propose novel online learning algorithms that can achieve constant regret on partial states depending on the task similarity and near-optimal regret when the two tasks are dissimilar. Furthermore, for low-tier tasks, these algorithms keep near-optimal regret at very little cost. The authors also study the scenario when multiple low-tier tasks are present and propose a novel transfer source selection algorithm that can gather knowledge from all low-tier tasks and produce benefits on a much larger state-action space.

STRENGTHS:
* The regret analysis of the robust tiered multi-armed bandit (MAB) models was very thorough.

WEAKNESSES:
* No experiments that compared performance with other tiered RL algorithms
* It would have nice to have had the related work in the manuscript instead of as a supplement.

QUESTIONS:
* What type of data have others used on this problem or similar RL problems? Can it be leveraged in this setting as well?

LIMITATIONS:
* The authors did not address the fact that they did not perform any experiments with either synthetic or real data on their model.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a robust parallel knowledge transfer reinforcement learning algorithm for single or multiple source tasks without knowledge on model similarity using the previously defined Tiered Reinforcement Learning framework. The paper remove the limitation on prior knowledge about the task similarity to generalize the framework. The main contribution is three-fold: 1) Establish necessary conditions for lower regret bound, 2) propose robust parallel transfer algorithms for reinforcement learning and its special case, multi-armed bandits, and 3) describe a new source task selection mechanism that guarantees constant regret for larger space of state-action pairs using multiple low-tier tasks.\n\nSTRENGTHS:\nOriginality. Removing the assumption on task similarity poses new challenges to guarantee lower regret bounds that are not trivial. The new algorithm to solve the general knowledge transfer require learning whether low-tier and high-tier tasks are similar from observed data and balance between exploration and exploration from the low-tier model at the same time. The lower bounds from simplifying the general case simplify to the prior work under the similarity conditions between low- and high-tier tasks. Therefore, the lower bounds are sound. To extend the approach for the high-tier task to leverage multiple low-tier tasks, the authors describe a method called “trust till failure” that still guarantees lower bounds on regret.\n\nQuality & Clarity. The contribution and novelty is well defined. The definitions and equations are sound.\n\nSignificance. The proposed approach generalizes the previous work to a larger class of problems while still maintaining lower bounds on regret minimization.\n\nWEAKNESSES:\nOriginality & Significance. To remove the limitation of task similarity for generalization is not trivial. However, the motivation for generalization is lacking with few references to potential robotics applications. The paper can benefit from providing an illustrative example or toy problems to motivate new class of problems the proposed method can be applied to. \n\nQuality. I am assuming the authors made a mistake and uploaded the incomplete paper. The related work is supposed to be in the Appendix but there are no Appendices attached to the submitted paper. Without these Appendices, it is difficult to compare how the new method compared to prior work. There are several grammatical errors and the conclusion section does not talk about the limitations of the current approach. The future work is also in Appendix which is not included in the paper.\n\nClarity. It is not clear how parallel knowledge transfer is any different from meta-learning and/or why it is not a special case of meta-learning. It will be good to discuss when the learning methods overlap or differ under which conditions to better motivate the approach. Specifically, for the use case of multiple low-tier learners with a single high-tier learner.\n\nQUESTIONS:\nPlease provide the appendices to complete the evaluation of the work on soundness and its relation to prior work.\n\nLIMITATIONS:\nThe authors do not provide limitations of the proposed approach. There are no societal issues related to this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper studies the tiered reinforcement learning setting, a transfer learning setting where the goal is to transfer information from a low tier task to a higher tier one while learning both to solve both tasks in parallel.
Contrary to prior work the author do not assume that both tasks share the same rewards and dynamics, they show it is still possible to benefit from the low tier task to learn the higher tier one. The author also extend their work by considering multiple low tier tasks and present selection mechanism which can gather information the different tasks.

STRENGTHS:
Overall this is a solid paper, while technical the paper is well written and properly organized.
Removing the assumption that the low and high tier sources share the same dynamics and rewards makes this setting much more applicable and interesting.
The setup with multiple low tier tasks is also valuable could have many applications.

WEAKNESSES:
It would have been interesting to add empirical evaluations of the proposed algorithms to understand their actual performance.

QUESTIONS:
In the multiple source tier setting is there a point where adding more tasks with a low similarity with the high tier tasks can hurt performance?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends the work of Huang et al (2022) on Tiered Reinforcement Learning (where the objective is to transfer knowledge from the low tier (risk-tolerant) source task to a high-tier (risk-averse) target task while solving the two two tasks in parallel) by relaxing the assumptions of identical reward and transition functions in the source and target tasks.\nThe paper first identifies ‘Optimal Value Dominance’ a necessary condition to keep the source algorithm near-optimal while achieving provable benefits for the regret in the high-tier algorithm when Low and High tier tasks are similar.\nThe paper then proposes algorithms for tiered multiarmed bandit and tiered reinforcement learning with regret lower bounds depending on the similarity between the low- and high-tier tasks, with an improved lower bound for the case when the tasks are the same compared to Huang et al (2022).\nFinally, the paper also considers a case when there are multiple low-tier tasks, in which the most similar states to the high-tier task are chosen from any of the low tier tasks for an additional log W factor in regret.\n\nSTRENGTHS:\nOriginality\n\nThe paper extends the framework (multiple low-tier tasks) of and relaxes some of the assumptions (non-identical reward and transition function among the low- and high-tier tasks) of a previous paper on Tiered RL.\n\nQuality\n\nThe paper is mostly well-written, clear and concise. The paper motivates the problem setting well in the introduction, and places it well in the literature. The concepts are introduced and explained in a logical order.\n\nClarity\n\nThe paper has a similar structure to Huang et al (2022). It is mostly clear and easy to follow, although certain parts do feel a bit rushed. (I will mention them in the Weaknesses part.)\n\nSignificance\n\nThe paper contributes a few findings in Tiered RL that could be useful to the community: it proves a tighter lower bound for the case when the lower and higher tier tasks are the same, and proves lower bounds for the case when the reward and transition functions differ depending on the difference in the gaps. It also introduces to setting when there are multiple low-tier tasks.\n\nWEAKNESSES:\nThe work in this paper results from a minor relaxation of certain assumption in Huang et al (2022), and a lot of possible extensions and improvements are left for future work.\n\nThe RL results depend on a hyperparameter, lambda that needs to be chosen wisely.\n\nMinor typos/grammar/style issues that did not affect the rating of this paper:\n\nLines 37, 41 and 52: “In [13]” is considered bad style, either write Author et al (year) [13], or better yet just cite the claim.\n\nLines 62: conceptions -> findings\n\nLines 67-68, starting with while: that subsentence makes no sense, please rephrase\n\nThe sentence on lines 125-127 seems like it belongs to the Frequently Used Notations subsection, not to 2.1\n\nLine 165: no need for ‘actually’\n\nLine 166: ‘leave it to the future work’ -> ‘leave it for future work’\n\nLine 176: remove ‘kind of’ or replace with ‘may [explain]’\n\nLine 177 is -> are, literatures -> papers/works/publications\n\nQUESTIONS:\nWhat do you mean by ‘back-propagation process of value iteration’ (line 255)? I am familiar with the concepts of backpropagation and value iteration individually, but their combination is rather ambiguous to me.\n\nHow would one go about choosing the lambda hyperparameter before beginning to solve the tasks?\n\nLIMITATIONS:\nThe authors are upfront about the limitations, such as the unique optimal policy assumption, and mention in the appendix that the avoidance of the lower bound knowledge about the minimal gap would be beneficial. Concerns for negative societal impact in this theoretical paper are not applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}