SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 3: Robust Tiered MAB with Multiple Source Tasks
1 Initilize:  $\alpha >2$ $N_{\mathrm{Lo}}^{1}(i)$ $N_{\mathrm{Hi}}^{1}(i)$ $\widehat{\mu}_{\mathrm{Lo}}^{1}(i)$ $\widehat{\mu}_{\mathrm{Hi}}^{1}(i)\gets 0$ $\forall i\in \mathcal{A}$ $f(k):= 1 + 16TA^2 (k + 1)^2$    
2 Pull each arm at the beginning  $A$  iterations. Set  $w^{A}\leftarrow \mathrm{Null}$    
3 for  $k = A + 1,2,\dots,K$  do   
4 for  $w = 1,2\dots ,W$  do   
5  $\overline{\mu}_{\mathrm{Lo},w}^{k}(i)\gets \widehat{\mu}_{\mathrm{Lo},w}^{k}(i) + \sqrt{\frac{2\alpha\log f(k)}{N_{\mathrm{Lo},w}(i)}},\quad \overline{\pi}_{\mathrm{Lo},w}^{k}\gets \arg \max_{i}\overline{\mu}_{\mathrm{Lo},w}^{k}(i),\quad \pi_{\mathrm{Lo},w}^{k}\gets \overline{\pi}_{\mathrm{Lo},w}^{k}.$    
6  $\underline{\mu}_{\mathrm{Lo},w}^{k}(i)\gets \widehat{\mu}_{\mathrm{Lo},w}^{k}(i) - \sqrt{\frac{2\alpha\log f(k)}{N_{\mathrm{Lo},w}(i)}},\quad \underline{\pi}_{\mathrm{Lo},w}^{k}\gets \arg \max_{i}\underline{\mu}_{\mathrm{Lo},w}^{k}(i).$    
7 end   
8  $\overline{\mu}_{\mathrm{Hi}}^{k}(i)\gets \widehat{\mu}_{\mathrm{Hi}}^{k}(i) + \sqrt{\frac{2\alpha\log f(k)}{N_{\mathrm{Hi}}^{k}(i)}},\quad \overline{\pi}_{\mathrm{Hi}}^{k}\gets \arg \max_{i}\overline{\mu}_{\mathrm{Hi}}^{k}(i).$    
9  $\mathcal{I}^k\gets \{w\in [W]|\underline{\mu}_{\mathrm{Lo},w}^k (\underline{\pi}_{\mathrm{Lo},w}^k)\leq \overline{\mu}_{\mathrm{Hi}}^k (\underline{\pi}_{\mathrm{Lo},w}^k) + \varepsilon \text{and} N_{\mathrm{Lo},w}^k (\underline{\pi}_{\mathrm{Lo},w}^k) > k / 2\}$    
10 if  $\mathcal{I}^k = \emptyset$  then  $w^{k}\gets \mathrm{Null}$  ，  $\pi_{\mathrm{Hi}}^{k}\gets \overline{\pi}_{\mathrm{Hi}}^{k}$  .   
11 else   
12 if  $w^{k - 1}\neq Nu11$  and  $w^{k - 1}\in \mathcal{I}^k$  then  $w^{k}\gets w^{k - 1}$  ，  $\pi_{\mathrm{Hi}}^{k}\gets \underline{\pi}_{\mathrm{Lo},w}^{k}$  .   
13 else if  $w^{k - 1}\neq Nu11$  and  $\exists w\in \mathcal{I}^k$  such that  $\pi_{\mathrm{Hi}}^{k - 1} = \arg \max_{i}N_{Lo,w}^{k}(i)$  then   
14  $|w^{k}\gets w,\quad \pi_{\mathrm{Hi}}^{k}\gets \underline{\pi}_{\mathrm{Lo},w}^{k}$    
15 end   
16 else  $w^{k}\sim \operatorname {Unif}(\mathcal{I}^{k})$  ，  $\pi_{\mathrm{Hi}}^{k}\gets \underline{\pi}_{\mathrm{Lo},w^{k}}^{k}$  .   
17 end   
18 Interact with  $M_{\mathrm{Hi}} / \{M_{\mathrm{Lo}},\}_{w = 1}^{W}$  by  $\pi_{\mathrm{Hi}}^{k} / \{\pi_{\mathrm{Lo},w}^{k}\}_{w = 1}^{W}$  .   
19 Update  $\{N_{\mathrm{Lo},w}^{k + 1}\}_{w = 1}^W,N_{\mathrm{Hi}}^{k + 1}$  and empirical mean  $\{\widehat{\mu}_{\mathrm{Lo},w}^{k + 1}\}_{w = 1}^W,\widehat{\mu}_{\mathrm{Hi}}^{k + 1}$  for each arm.
For the lack of space, in the following, we only analyze the bandit setting to explain the key idea of our task selection strategy. For the RL setting, we defer to Appx. F the algorithm Alg. 7, detailed version of Thm. 5.2 (Thm. D.16), definition of transferable set  $\mathcal{C}_h^{\lambda,[W]}$  (Def. F.2), and technical details.
Algorithm Design and Proof Sketch for Bandit Setting The algorithm is provided in Alg. 3. Comparing with Alg. 1 in single task setting, the main difference is the task selection strategy from line 9 to line 17. We first examine each source task with a checking event similar to single task setting, and collect those feasible tasks passing the test to  $\mathcal{I}^k$ . Intuitively, for those  $M_{\mathrm{Lo},w^*}$  close to  $M_{\mathrm{Lo}}$ , we expect  $M_{\mathrm{Lo},w} \in \mathcal{I}^k$  holds almostly for arbitrary  $k > 0$ , while for the other  $M_{\mathrm{Lo},w'}$ , if it takes the position of  $w^k$ , following  $M_{\mathrm{Lo},w'}$  will reduce the uncertainty and it will be ruled out from  $\mathcal{I}^k$ , eventually. So we expect  $w^k$  can "escape" from dissimilar source tasks but be absorbed to the similar task if exists. Therefore, if  $\mathcal{I}^k$  is empty,  $\mathrm{Alg}^{\mathrm{Hi}}$  will do exploration by itself. Otherwise, we choose one from  $\mathcal{I}^k$  to transfer the action until it fails on the checking event. However, for any  $\varepsilon$  the algorithm chosen, those "marginally similar" source tasks (denoted as  $M_{\mathrm{Lo},\tilde{w}}$ ), which are  $\varepsilon'$ -close to  $M_{\mathrm{Hi}}$  for some  $\varepsilon'$  only slightly larger than  $\varepsilon$ , may cause some trouble. Because the checking event
will finally eliminate  $M_{\mathrm{Lo}, \tilde{w}}$  since they are not  $\varepsilon$ -close, but it may occupy the position  $w^k$  for a long time before elimination, especially when  $\varepsilon'$  is extremely close to  $\varepsilon$ . After eliminating  $M_{\mathrm{Lo}, \tilde{w}}$ ,  $\mathrm{Alg}^{\mathrm{Hi}}$  needs to re-select one from  $\mathcal{I}^k$ . Now since other sub-optimal arms in  $M_{\mathrm{Hi}}$  haven't been chosen for a long time and the confidence level  $\delta_k = O(1 / k^\alpha)$  is decreasing,  $\mathcal{I}^k$  will include those dissimilar tasks again, which causes difficulty to identify the true similar task. To solve this issue, once the previous trusted task fails, we give priority to the task recommending the same action as the previous one (line 14). As a result, since  $M_{\mathrm{Lo}, \tilde{w}}$  and  $M_{\mathrm{Lo}, w^*}$  share the optimal action, after the elimination of  $M_{\mathrm{Lo}, \tilde{w}}$ , we can expect  $w^k$  to only switch among those tasks  $M_{\mathrm{Lo}, w}$  with  $\pi_{\mathrm{Lo}, w}^* = \pi_{\mathrm{Hi}}^*$ . We highlight this technical novelty to Lem. 5.3 below, and defer all the proofs to Appx. E.
Lemma 5.3. [Absorbing to Similar Task] Under Assum.  $A$ ,  $B$  and  $C$ , there exists a constant  $c^*$ , s.t., if there exists at least one  $w^* \in [W]$  such that  $M_{Lo,w^*}$  is  $\frac{\widetilde{\Delta}_{\min}}{4}$ -close to  $M_{Hi}$ , by running Alg. 3 with  $\varepsilon = \frac{\widetilde{\Delta}_{\min}}{4}$  and  $\alpha > 2$ , for any  $k \geq k^* := c^* \frac{\alpha A}{\Delta_{\min}^2} \log \frac{\alpha AW}{\Delta_{\min}}$ , we have  $\operatorname*{Pr}(\pi_{Hi}^k \neq i_{Hi}^*) = O(\frac{A}{k^{2\alpha - 2}})$ .
In this section, we evaluate our most representative algorithm, Alg. 7, in multiple source tasks setting.
Experiments Setting<sup>3</sup> We set  $S = A = 3$  and  $H = 5$ . The details for construction of source and target tasks are deferred to Appx. G. We adapt StrongEuler in [23] as online learning algorithm to solve source tasks, and use the bonus function in [23] as the bonus function in our Alg. 7. We evaluate our algorithm when  $W = 0, 1, 2, 5$ , where  $W = 0$  means the high-tier task  $M_{\mathrm{Hi}}$  is simply solved by normal online learning method (StrongEuler) without any parallel knowledge transfer. We choose  $\lambda = 0.3 \approx 1 / S$  in Alg. 7, and in the MDP instance we test, across all  $S \cdot H = 15$  states, for  $W = 1, 2, 5$ , the number of transferable states would be 6, 9 and 13, respectively.
We choose iteration number  $K = 1e7$ , where we start the transfer since  $k = 5e5$  to avoid large "burn-in" terms. As we can see, after the transfer starts, the regret in target task will suddenly increase for a while, because the target task has to make some mistakes and learn from it as a result of the model uncertainty. However, because of our algorithm design, the negative transfer will terminate after a very short period. As predicted by our theory, by adding more and more source tasks which can introduce new transferable states, the target task will suffer less and less regret.
Figure 1: Regret in the Target Task given Multiple Source Tasks We report the result when  $W$  source tasks are available with  $W = 0,1,2,5$ . The shadows indicate  $96\%$  confidence interval.
In this paper, we study how to do robust parallel transfer RL when single or multiple source tasks are available, without knowledge on models' similarity. The possible future directions include relaxing assumptions, better strategies to leveraging multiple source tasks, and identifying mild structural assumptions allowing for more aggressive transfer, and we defer to Appx. A.4 for more details.
$^{3}$ Code is available at https://github.com/jiaweiihuang/Robust-Tiered-RL
The authors would like to thank Andreas Krause for valuable discussion. The work is supported by ETH research grant and Swiss National Science Foundation (SNSF) Project Funding No. 200021-207343.
[1] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pages 10-4, 2019.
[2] Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263-272. PMLR, 2017.
[3] Emma Brunskill and Lihong Li. Sample complexity of multi-task reinforcement learning, 2013.
[4] Jinglin Chen and Nan Jiang. Offline reinforcement learning under value and density-ratio realizability: the power of gaps. arXiv preprint arXiv:2203.13935, 2022.
[5] HeeSun Choi, Cindy Crump, Christian Duriez, Asher Elmquist, Gregory Hager, David Han, Frank Hearl, Jessica Hodgins, Abhinandan Jain, Frederick Leve, et al. On the use of simulation in robotics: Opportunities, challenges, and suggestions for moving forward. Proceedings of the National Academy of Sciences, 118(1):e1907856118, 2021.
[6] Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning, 2017.
[7] Christoph Dann, Teodor Vanislavov Marinov, Mehryar Mohri, and Julian Zimmert. Beyond value-function gaps: Improved instance-dependent regret bounds for episodic reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=CU8qQMhB3dh.
[8] Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, Jan Peters, et al. Sharing knowledge in multi-task deep reinforcement learning. In 8th International Conference on Learning Representations, {ICLR} 2020, Addis Ababa, Ethiopia, April 26-30, 2020, pages 1-11. OpenReview.net, 2020.
[9] Noah Golowich and Ankur Moitra. Can q-learning be improved with advice? In Conference on Learning Theory, pages 4548-4619. PMLR, 2022.
[10] Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham M. Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=D-X3kH-BkpN.
[11] Jiachen Hu, Xiaoyu Chen, Chi Jin, Lihong Li, and Liwei Wang. Near-optimal representation learning for linear bandits and linear rl. In International Conference on Machine Learning, pages 4349-4358. PMLR, 2021.
[12] Jiawei Huang, Jinglin Chen, Li Zhao, Tao Qin, Nan Jiang, and Tie-Yan Liu. Towards deployment-efficient reinforcement learning: Lower bound and optimality. arXiv preprint arXiv:2202.06450, 2022.
[13] Jiawei Huang, Li Zhao, Tao Qin, Wei Chen, Nan Jiang, and Tie-Yan Liu. Tiered reinforcement learning: Pessimism in the face of uncertainty and constant regret. arXiv preprint arXiv:2205.12418, 2022.
[14] Nan Jiang and Jiawei Huang. Minimax value interval for off-policy evaluation and policy optimization. Advances in Neural Information Processing Systems, 33:2747-2758, 2020.
[34] Haike Xu, Tengyu Ma, and Simon Du. Fine-grained gap-dependent bounds for tabular mdps via adaptive multi-step bootstrap. In Conference on Learning Theory, pages 4438-4472. PMLR, 2021.
[35] Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pages 2730–2775. PMLR, 2022.
[36] Chicheng Zhang and Zhi Wang. Provably efficient multi-task reinforcement learning with model transfer. Advances in Neural Information Processing Systems, 34:19771-19783, 2021.
[37] Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprint arXiv:2009.07888, 2020.