SlowGuess/NeurIPS_2023_OCR huggingface dataset:
After learning an actor in  $\mathcal{M}_k^1$  with the experimental setup described in Section B, we train KIAN for  $\mathcal{M}_k^2$  by initializing its external knowledge keys with the knowledge keys learned in  $\mathcal{M}_k^1$ . These external knowledge keys remain fixed when learning  $\mathcal{M}_k^2$ , and all other components of KIAN are learned from scratch. This setup allows us to test the efficacy of reusing learned knowledge keys across different tasks. All actor components of RL, KoGuN, and A2T are learned from scratch in  $\mathcal{M}_k^2$ . This is because RL does not incorporate any external knowledge, and the model architectures of KoGuN and A2T do not allow changing the number and order of knowledge policies. The hyperparameters of learning  $\mathcal{M}_k^1$  and  $\mathcal{M}_k^2$  are the same as those listed in Section B.2.4. The external knowledge policies used for each experiment are detailed as follows:
- Dynamic-Obstacles-16x16  $\rightarrow$  LavaCrossingS9N2. Reuse the knowledge embedding of "go to the goal"; reuse the knowledge embedding of "do not hit balls" as the fixed knowledge embedding of "do not hit".
- Unlock  $\rightarrow$  DoorKey-8x8. Reuse the knowledge embeddings of "get the key" and "open the door".
- Dynamic-Obstacles-16x16  $\rightarrow$  MultiRoom-N4-S5. Reuse the knowledge embedding of "go to the goal".
- DoorKey-8x8  $\rightarrow$  MultiRoom-N4-S5. Reuse the knowledge embedding of "go to the goal"; reuse the knowledge embedding of "open the door" as the fixed knowledge embedding of "open the unlocked door".
- Dynamic-Obstacles-16x16  $\rightarrow$  KeyCorridorS3R2. Reuse the knowledge embedding of "go to the goal" as the fixed knowledge embedding of "pick up the ball".
- DoorKey-8x8  $\rightarrow$  KeyCorridorS3R2. Reuse the knowledge embedding of "get the key"; reuse the knowledge embedding of "open the door"; reuse the knowledge embedding of "go to the goal" as the fixed knowledge embedding of "pick up the ball".
C.2 OpenAI-Robotic Environments
After learning an actor and a critic in  $\mathcal{M}_k^1$  with the experimental setup described in Section B, we initialize the networks for  $\mathcal{M}_k^2$  as follows:
- RL: The actor and critic of  $\mathcal{M}_k^2$  are initialized with that of  $\mathcal{M}_k^1$ . These networks will be updated when learning in  $\mathcal{M}_k^2$ .
- KoGuN: Only the critic of  $\mathcal{M}_k^2$  is initialized with that of  $\mathcal{M}_k^1$ . The actor is learned from scratch in  $\mathcal{M}_k^2$ . The actor and critic will be updated when learning in  $\mathcal{M}_k^2$ .
- A2T: The critic of  $\mathcal{M}_k^2$  is initialized with that of  $\mathcal{M}_k^1$ . The inner policy and attention network are learned from scratch in  $\mathcal{M}_k^2$ . All networks will be updated when learning in  $\mathcal{M}_k^2$ .
- KIAN: The external knowledge keys, query, and critic of  $\mathcal{M}_k^2$  are initialized with that of  $\mathcal{M}_k^1$ . Note that the external knowledge keys of  $\mathcal{M}_k^2$  include the internal and external knowledge keys from  $\mathcal{M}_k^1$ . The inner policy and inner knowledge key are learned from scratch in  $\mathcal{M}_k^2$ . The external knowledge keys remain fixed, while other networks will be updated when learning in  $\mathcal{M}_k^2$ .
The hyperparameters of learning  $\mathcal{M}_k^1$  are the same as those listed in Section B.3.4. When learning  $\mathcal{M}_k^2$ , the hyperparameters changed are listed as follows: The training timesteps are 0.4M. The learning rates are  $7.5 \times 10^{-4}$  and  $10^{-3}$  for FetchPush and FetchPickAndPlace respectively.
D Broader Impact and Future Research Directions
The KGRL framework presented in this work aims to enhance an agent's ability to learn from external policies. These policies encompass not only sub-optimal strategies to task completion but also regulatory policies that emphasize safety constraints and ethical behaviors. Being able to incorporate safety- and ethics-oriented policies gives KGRL the potential to significantly influence an artificial agent's behavior, promoting enhanced safety and social acceptability. These aspects have gained substantial attention in the field of RL [38-42, 46, 48], underscoring their importance in contemporary research.
Moving forward, there are several research directions in KGRL that are worth exploring. First, fusing knowledge policies with different state and action spaces enables efficient learning across a broader range of applications. Second, integrating regulatory policies that enforce strict constraints during learning and inference stages can ensure adherence to safety and ethical considerations. Lastly, addressing complex relationships among different policies, such as conditional dependence and conflicts, allows an agent to efficiently navigate through a large and diverse set of knowledge policies. We hope these directions have the potential to inspire future studies in KGRL.
E Other Experimental Results
We provide the standard deviation for the experiments of generalizability in Table 3-5.
Train in Test in	Empty-Random-5x5	DoorKey-5x5
6x6	8x8	16x16	8x8	16x16
RL [10, 29]	.88±.06	.71±.20	.45±.35	.29±.16	.08±.10
RL+BC [21]	.87±.03	.60±.14	.24±.17	.40±.01	.09±.08
KoGuN [36]	.94±.01	.83±.03	.53±.11	.77±.09	.35±.08
A2T [27]	.92±.01	.78±.11	.51±.30	.53±.09	.11±.03
KIAN (ours)	.96±.02	.91±.01	.93±.02	.76±.01	.42±.08
Table 3: MiniGrid Zero-Shot S2C Experiments.
Train in Test in	Push	Slide	Pick-and-Place
5x	10x	5x	10x	5x	10x
RL [10, 29]	.87±.05	.52±.11	.45±.07	.17±.05	.34±.54	.27±.44
RL+BC [21]	.89±.02	.60±.09	.44±.10	.16±.02	.34±.55	.30±.50
KoGuN [36]	.63±.49	.43±.32	.55±.07	.18±.04	.32±.52	.24±.38
A2T [27]	.03±.00	.05±.00	.00±.00	.01±.00	.01±.00	.06±.00
KIAN (ours)	.93±.05	.70±.02	.42±.10	.15±.04	.92±.00	.72±.03
Table 4: OpenAI-Robotics Zero-Shot S2C Experiments.
Train in Test in	DoorKey-5x5 Empty-Random	DoorKey-8x8 Unlock	DoorKey5x5	Pick-and-Place Reach	Push Push	Push Reach	Slide Push
RL [10, 29]	.83±.07	.92±.01	.93±.01	.80±.45	.31±.19	.16±.10	.09±.04
RL+BC [21]	.85±.05	.87±.03	.93±.01	.80±.45	.31±.19	.16±.10	.09±.04
KoGuN [36]	.90±.02	.91±.01	.93±.01	.45±.37	.05±.02	.20±.07	.07±.02
A2T [27]	.84±.04	.92±.01	.93±.00	.01±.00	.05±.00	.20±.45	.05±.00
KIAN (ours)	.91±.01	.94±.01	.95±.00	1.0±.00	.30±.04	.24±.06	.13±.02
Table 5: Zero-Shot C2S Experiments.
F Effects of Extensive External Knowledge Set with Irrelevant Policies
In order for KGRL agents to effectively leverage an external knowledge set, it is imperative that they can (1) distinguish which external policies are less related to the given task and (2) efficiently navigate through an
extensive collection of external policies. Failure to accomplish these objectives in a timely manner could result in suboptimal performance, potentially even inferior to that of an RL agent. Under such circumstances, integrating external policies into the learning process becomes impractical.
In this section, we examine the following two aspects of KIAN: (1) the ability of KIAN to rapidly acquire valuable strategies, even in the presence of random or irrelevant policies within the external knowledge set, and (2) the impact of KIAN's performance as the size of the external knowledge set increases.
Figure 6 shows the learning curves of PPO and  $\mathrm{PPO + KIAN}$  considering various numbers of external policies: 2 relevant, 4 (2 relevant  $+2$  irrelevant), and 6 (2 relevant  $+4$  irrelevant) for the MiniGrid Unlock task. The results indicate that including more irrelevant knowledge policies leads to a marginal decline in performance, but the agents consistently achieve high rewards with minimal variances. This minor decline in performance aligns with our expectations since the agents need to (1) navigate through a more extensive set of external policies and (2) distinguish and disregard policies that do not contribute to solving the task. Therefore, when the external knowledge set is very large, KGRL methods, such as KoGuN, A2T, and KIAN, do not guarantee superior efficiency over RL methods. Efficiently harnessing a large external policy set remains an avenue of future research and exploration.
Figure 6: Learning curves of PPO and PPO+KIAN for the MiniGrid Unlock task, with external knowledge sets that include irrelevant policies.
[38] Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3387-3395, 2019.
[39] Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. Advances in neural information processing systems, 31, 2018.
[40] Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statistics, pages 3304-3312. PMLR, 2021.
[41] Adrien Ecoffet and Joel Lehman. Reinforcement learning under moral uncertainty. In International conference on machine learning, pages 2926-2936. PMLR, 2021.
[42] Javier Garcia and Fernando Fernández. Safe exploration of state and action spaces in reinforcement learning. Journal of Artificial Intelligence Research, 45:515-564, 2012.
[43] John R Hershey and Peder A Olsen. Approximating the kullback leibler divergence between gaussian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP'07, volume 4, pages IV-317. IEEE, 2007.
[44] Isidore I Hirschman. A note on entropy. American journal of mathematics, 79(1):152-156, 1957.
[45] Marco F Huber, Tim Bailey, Hugh Durrant-Whyte, and Uwe D Hanebeck. On entropy approximation for gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pages 181–188. IEEE, 2008.
[46] Puze Liu, Davide Tateo, Haitham Bou Ammar, and Jan Peters. Robot reinforcement learning on the constraint manifold. In Conference on Robot Learning, pages 1357-1366. PMLR, 2022.
[47] Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3): 379-423, 1948.
[48] Yueh-Hua Wu and Shou-De Lin. A low-cost ethics shaping approach for designing reinforcement learning agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.