SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \pi_ {2} \right\| _ {\infty} = \max  _ {i \in \{1, \dots , d _ {a} \}} \left| \pi_ {2, i} \right| = \max  _ {i \in \{1, \dots , d _ {a} \}} \left| \frac {1}{d _ {a}} + \varepsilon_ {2, i} \right| > \frac {1}{d _ {a}}. \tag {26}
$$
A.2 Proof of Proposition 4.2
In continuous action space, the final policy fused by equation (3) is a mixture of normal distributions, also known as a Gaussian mixture. However, in general, the entropy of a Gaussian mixture does not have a closed form [45]. Instead of analyzing the entropy of  $\pi$ , we analyze the variance of  $\pi$  for  $d_{a} = 1$  in maximum entropy KGRL since for any probability density function of a real-valued random variable, Shannon's inequality for entropy and variance [44, 47] specifies
$$
H (\pi) \leq \frac {1}{2} \ln \left(2 p \sigma^ {2}\right) + \frac {1}{2}, \tag {27}
$$
where  $\sigma^2\in \mathbb{R}_{\geq 0}$  is the variance of  $\pi$  and  $p\approx 3.14159$
$$
\pi = \hat {w} _ {1} \pi_ {1} + \hat {w} _ {2} \pi_ {2}, \text {w h e r e} \pi_ {j} = \mathcal {N} \left(\mu_ {j}, \sigma_ {j} ^ {2}\right), \hat {w} _ {j} \geq 0 \forall j \in \{1, 2 \}, \text {a n d} \hat {w} _ {1} + \hat {w} _ {2} = 1. \tag {28}
$$
Proof. Let  $a$  be a continuous random variable with the probability density function being  $\pi(\cdot|\mathbf{s})$ . Then its first and second moments are
$$
\begin{array}{l} \mathbb {E} _ {a \sim \pi} [ a ] = \int a \pi (\cdot | \mathbf {s}) d a (29) \\ = \int a \left(\hat {w} _ {1} \pi_ {1} (a | \mathbf {s}) + \hat {w} _ {2} \pi_ {2} (a | \mathbf {s})\right) d a (30) \\ = \hat {w} _ {1} \int a \pi_ {1} (a | \mathbf {s}) d a + \hat {w} _ {2} \int a \pi_ {2} (a | \mathbf {s}) d a (31) \\ = \hat {w} _ {1} \mu_ {1} + \hat {w} _ {2} \mu_ {2} (32) \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {a \sim \pi} \left[ a ^ {2} \right] = \int a ^ {2} \pi (\cdot | \mathbf {s}) d a (33) \\ = \int a ^ {2} \left(\hat {w} _ {1} \pi_ {1} (a | \mathbf {s}) + \hat {w} _ {2} \pi_ {2} (a | \mathbf {s})\right) d a (34) \\ = \hat {w} _ {1} \int a ^ {2} \pi_ {1} (a | \mathbf {s}) d a + \hat {w} _ {2} \int a ^ {2} \pi_ {2} (a | \mathbf {s}) d a (35) \\ = \hat {w} _ {1} \left(\mu_ {1} ^ {2} + \sigma_ {1} ^ {2}\right) + \hat {w} _ {2} \left(\mu_ {2} ^ {2} + \sigma_ {2} ^ {2}\right). (36) \\ \end{array}
$$
Equation (36) holds since  $\sigma_{j}^{2} = \mathbb{E}_{a_{j}\sim \pi_{j}}[a_{j}^{2}] - \left(\mathbb{E}_{a_{j}\sim \pi_{j}}[a_{j}]\right)^{2},\forall j\in \{1,2\}$ . The variance of  $a$  thus becomes
$$
\begin{array}{l} \mathbb {V} _ {a \sim \pi} [ a ] = \mathbb {E} _ {a \sim \pi} [ a ^ {2} ] - (\mathbb {E} _ {a \sim \pi} [ a ]) ^ {2} (37) \\ = \hat {w} _ {1} \left(\mu_ {1} ^ {2} + \sigma_ {1} ^ {2}\right) + \hat {w} _ {2} \left(\mu_ {2} ^ {2} + \sigma_ {2} ^ {2}\right) - \left(\hat {w} _ {1} \mu_ {1} + \hat {w} _ {2} \mu_ {2}\right) ^ {2} (38) \\ = \hat {w} _ {1} \sigma_ {1} ^ {2} + \hat {w} _ {2} \sigma_ {2} ^ {2} + \hat {w} _ {1} (1 - \hat {w} _ {1}) \mu_ {1} ^ {2} + \hat {w} _ {2} (1 - \hat {w} _ {2}) \mu_ {2} ^ {2} - 2 \hat {w} _ {1} \hat {w} _ {2} \mu_ {1} \mu_ {2} (39) \\ = \hat {w} _ {1} \sigma_ {1} ^ {2} + \hat {w} _ {2} \sigma_ {2} ^ {2} + \hat {w} _ {1} \hat {w} _ {2} \left(\mu_ {1} ^ {2} + \mu_ {2} ^ {2} - 2 \mu_ {1} \mu_ {2}\right) (40) \\ = \hat {w} _ {1} \sigma_ {1} ^ {2} + \hat {w} _ {2} \sigma_ {2} ^ {2} + \hat {w} _ {1} \hat {w} _ {2} (\mu_ {1} - \mu_ {2}) ^ {2}. (41) \\ \end{array}
$$
According to equation (41), if  $\sigma_1^2\gg \sigma_2^2$  and  $\sigma_1^2\gg (\mu_1 - \mu_2)^2$ , maximizing  $\mathbb{V}_{a\sim \pi}[a]$  leads to  $\hat{w}_1\rightarrow 1$ .
A.3 Proof of Proposition 4.3
Proof. According to equation (41), if  $\hat{w}_1, \hat{w}_2, \sigma_1^2$ , and  $\sigma_2^2$  are fixed, maximizing  $\mathbb{V}_{a \sim \pi}[a]$  results in maximizing  $(\mu_1 - \mu_2)^2$ , hence increasing the distance between  $\mu_1$  and  $\mu_2$ .
A.4 Proof of Proposition 4.4
Before proving Proposition 4.4, we first show that KL divergence between a mixture of uniform distributions and a mixture of normal distributions is upper-bounded by a constant.
Proposition A.2 (KL divergence between a mixture of uniform distributions and a Gaussian mixture). Given a Gaussian mixture
$$
\pi (\cdot) = \sum_ {j = 1} ^ {m} \hat {w} _ {j} \mathcal {N} (\cdot ; \mu_ {j}, \sigma_ {j} ^ {2}), \tag {42}
$$
$$
w h e r e \quad \mathcal {N} (x; \mu_ {j}, \sigma_ {j} ^ {2}) = \frac {1}{\sqrt {2 p \sigma_ {j} ^ {2}}} \exp \frac {- (x - \mu_ {j}) ^ {2}}{2 \sigma_ {j} ^ {2}} \tag {43}
$$
and a mixture of  $m$  uniform distributions
$$
\hat {\pi} (\cdot) = \sum_ {j = 1} ^ {m} \hat {w} _ {j} \mathcal {U} (\cdot ; \mu_ {j} - \gamma_ {j}, \mu_ {j} + \gamma_ {j}), \tag {44}
$$
$$
w h e r e \quad \mathcal {U} (\cdot ; a, b) = \frac {1}{b - a} \quad a n d \quad \gamma_ {j} = \frac {1}{2 \mathcal {N} \left(\mu_ {j} ; \mu_ {j} , \sigma_ {j} ^ {2}\right)} \tag {45}
$$
for a real-valued random variable, the KL divergence between  $\hat{\pi} (\cdot)$  and  $\pi (\cdot)$  has an upper bound of  $\frac{p}{12}$ .
Proof. Since KL divergence  $D_{KL}(P\| Q)$  is convex in the pair  $(P,Q)$ ,  $D_{KL}(\hat{\pi}\| \pi)$  has the following upper bound [43]
$$
D _ {K L} (\hat {\pi} \| \pi) = D _ {K L} \left(\sum_ {j = 1} ^ {m} \hat {w} _ {j} \mathcal {U} \left(\cdot ; \mu_ {j} - \gamma_ {j}, \mu_ {j} + \gamma_ {j}\right) \| \sum_ {j = 1} ^ {m} \hat {w} _ {j} \mathcal {N} \left(\cdot ; \mu_ {j}, \sigma_ {j} ^ {2}\right)\right) \tag {46}
$$
$$
\leq \sum_ {j = 1} ^ {m} \hat {w} _ {j} D _ {K L} \left(\mathcal {U} \left(\cdot ; \mu_ {j} - \gamma_ {j}, \mu_ {j} + \gamma_ {j}\right) \| \mathcal {N} \left(\cdot ; \mu_ {j}, \sigma_ {j} ^ {2}\right)\right). \tag {47}
$$
For each  $j\in \{1,\ldots ,m\}$
$$
\begin{array}{l} D _ {K L} \left(\mathcal {U} \left(\cdot ; \mu_ {j} - \gamma_ {j}, \mu_ {j} + \gamma_ {j}\right) \| \mathcal {N} \left(\cdot ; \mu_ {j}, \sigma_ {j} ^ {2}\right)\right) (48) \\ = \int_ {\mu_ {j} - \gamma_ {j}} ^ {\mu_ {j} + \gamma_ {j}} \frac {1}{2 \gamma_ {j}} \ln \frac {\frac {1}{2 \gamma_ {j}}}{\frac {1}{\sqrt {2 p \sigma_ {j} ^ {2}}} \exp \left(- \frac {(x - \mu_ {j}) ^ {2}}{2 \sigma_ {j} ^ {2}}\right)} d x (49) \\ = \int_ {\mu_ {j} - \gamma_ {j}} ^ {\mu_ {j} + \gamma_ {j}} \frac {1}{2 \gamma_ {j}} \ln \frac {1}{2 \gamma_ {j}} d x - \int_ {\mu_ {j} - \gamma_ {j}} ^ {\mu_ {j} + \gamma_ {j}} \frac {1}{2 \gamma_ {j}} \left(\ln \frac {1}{\sqrt {2 p \sigma_ {j} ^ {2}}} - \frac {(x - \mu_ {j}) ^ {2}}{2 \sigma_ {j} ^ {2}}\right) d x (50) \\ = \ln \frac {1}{2 \gamma_ {j}} - \ln \frac {1}{\sqrt {2 p \sigma_ {j} ^ {2}}} + \int_ {\mu_ {j} - \gamma_ {j}} ^ {\mu_ {j} + \gamma_ {j}} \frac {(x - \mu_ {j}) ^ {2}}{4 \gamma_ {j} \sigma_ {j} ^ {2}} d x (51) \\ = \ln \frac {1}{2 \gamma_ {j}} - \ln \frac {1}{\sqrt {2 p \sigma_ {j} ^ {2}}} + \frac {\gamma_ {j} ^ {2}}{6 \sigma_ {j} ^ {2}} (52) \\ = \frac {p}{1 2}. (53) \\ \end{array}
$$
Equation (53) comes from substituting  $\gamma_{j}$  into equation (52).
Finally, the upper bound of  $D_{KL}(\hat{\pi}\| \pi)$  becomes
$$
\begin{array}{l} D _ {K L} (\hat {\pi} \| \pi) \leq \sum_ {j = 1} ^ {m} \hat {w} _ {j} D _ {K L} \left(\mathcal {U} \left(\because \mu_ {j} - \gamma_ {j}, \mu_ {j} + \gamma_ {j}\right) \| \mathcal {N} \left(\because \mu_ {j}, \sigma_ {j} ^ {2}\right)\right) (54) \\ = \sum_ {j = 1} ^ {m} \hat {w} _ {j} \frac {p}{1 2} = \frac {p}{1 2}. (55) \\ \end{array}
$$
Proposition 4.4 (Approximation of a mixture of normal distributions). If the following three inequalities hold for  $\mu_{t,in},\mu_{t,g_1},\ldots ,\mu_{t,g_n}$ , and  $a_{t,in}$ $\| \mu_{t,in} - \mu_{t,g_j}\| _2 <   \min \{\gamma_{t,in},\gamma_{t,g_j}\}$ $\| a_{t,in} - \mu_{t,in}\| _2 <   \min \{\gamma_{t,in},\gamma_{t,g_j}\}$ , and  $\| a_{t,in} - \mu_{t,g_j}\| _2 <   \gamma_{t,g_j}$ $\forall j\in \{1,\dots ,n\}$ , where  $\gamma_{t,in} = 1 / (2\pi_{in}(\mu_{t,in}|\mathbf{s}_t))$  and  $\gamma_{t,g_j} = 1 / (2\pi_{g_j}(\mu_{t,g_j}|\mathbf{s}_t))$ , then equation (9) for a real-valued action  $a_{t}$  sampled from KIAN can be approximated by
$$
\hat {w} _ {t, i n} \mathcal {U} \left(a _ {t}; \mu_ {t, i n} - \gamma_ {t, i n}, \mu_ {t, i n} + \gamma_ {t, i n}\right) + \sum_ {j = 1} ^ {n} \hat {w} _ {t, g _ {j}} \mathcal {U} \left(a _ {t}; \mu_ {t, i n} - \gamma_ {t, g _ {j}}, \mu_ {t, i n} + \gamma_ {t, g _ {j}}\right), \tag {56}
$$
$$
w h e r e \quad \mathcal {U} (\cdot ; a, b) = 1 / (b - a). \tag {57}
$$
Proof. Proposition A.2 shows that  $\pi (\cdot |\mathbf{s}_t)$  fused as equation (3) can be approximated by
$$
\hat {\pi} (\cdot | \mathbf {s} _ {t}) = \hat {w} _ {t, i n} \mathcal {U} (\cdot ; \mu_ {t, i n} - \gamma_ {t, i n}, \mu_ {t, i n} + \gamma_ {t, i n}) + \sum_ {j = 1} ^ {n} \hat {w} _ {t, g _ {j}} \mathcal {U} (\cdot ; \mu_ {t, g _ {j}} - \gamma_ {t, g _ {j}}, \mu_ {t, g _ {j}} + \gamma_ {t, g _ {j}}) \tag {58}
$$
with KL divergence being at most  $\frac{p}{12}$ , which is a constant.
Since any continuous action  $a_{t}$  outputted by KIAN belongs to  $\{a_{t,in},\mu_{t,g_1},\ldots ,\mu_{t,g_n}\}$  (Line 11 to 16 in Algorithm 1), if the three inequalities in the proposition statement hold, then for all  $a_{t}$
$$
\mathcal {U} \left(a _ {t}; \mu_ {t, i n} - \gamma_ {t, i n}, \mu_ {t, i n} + \gamma_ {t, i n}\right) = \frac {1}{2 \gamma_ {t , i n}}, \tag {59}
$$
$$
\mathcal {U} \left(a _ {t}; \mu_ {t, g _ {j}} - \gamma_ {t, g _ {j}}, \mu_ {t, g _ {j}} + \gamma_ {t, g _ {j}}\right) = \frac {1}{2 \gamma_ {t , g _ {j}}} \quad \forall j \in \{1, \dots , n \}, \text {a n d} \tag {60}
$$
$$
\mathcal {U} \left(a _ {t}; \mu_ {t, i n} - \gamma_ {t, g _ {j}}, \mu_ {t, i n} + \gamma_ {t, g _ {j}}\right) = \frac {1}{2 \gamma_ {t , g _ {j}}} \quad \forall j \in \{1, \dots , n \}. \tag {61}
$$
Therefore,  $\hat{\pi} (\cdot |\mathbf{s}_t)$  can be rewritten as
$$
\hat {\pi} (\cdot | \mathbf {s} _ {t}) = \hat {w} _ {t, i n} \mathcal {U} (\because \mu_ {t, i n} - \gamma_ {t, i n}, \mu_ {t, i n} + \gamma_ {t, i n}) + \sum_ {j = 1} ^ {n} \hat {w} _ {t, g _ {j}} \mathcal {U} (\because \mu_ {t, i n} - \gamma_ {t, g _ {j}}, \mu_ {t, i n} + \gamma_ {t, g _ {j}}). \tag {62}
$$
For any  $a_{t}\in \{a_{t,in},\mu_{t,g_{1}},\ldots ,\mu_{t,g_{n}}\}$
$$
\begin{array}{l} \hat {\pi} \left(a _ {t} \mid \mathbf {s} _ {t}\right) = \hat {w} _ {t, i n} \mathcal {U} \left(a _ {t}; \mu_ {t, i n} - \gamma_ {t, i n}, \mu_ {t, i n} + \gamma_ {t, i n}\right) + \sum_ {j = 1} ^ {n} \hat {w} _ {t, g _ {j}} \mathcal {U} \left(a _ {t}; \mu_ {t, i n} - \gamma_ {t, g _ {j}}, \mu_ {t, i n} + \gamma_ {t, g _ {j}}\right) (63) \\ = \hat {w} _ {t, i n} \frac {1}{2 \gamma_ {t , i n}} + \sum_ {j = 1} ^ {n} \hat {w} _ {t, g _ {j}} \frac {1}{2 \gamma_ {t , g _ {j}}} (64) \\ = \hat {w} _ {t, i n} \pi_ {i n} \left(\mu_ {t, i n} | \mathbf {s} _ {t}\right) + \sum_ {j = 1} ^ {n} \hat {w} _ {t, g _ {j}} \pi_ {g _ {j}} \left(\mu_ {t, g _ {j}} | \mathbf {s} _ {t}\right) (65) \\ \geq \hat {w} _ {t, i n} \pi_ {i n} \left(a _ {t, i n} \mid \mathbf {s} _ {t}\right) + \sum_ {j = 1} ^ {n} \hat {w} _ {t, g _ {j}} \pi_ {g _ {j}} \left(\mu_ {t, g _ {j}} \mid \mathbf {s} _ {t}\right). (66) \\ \end{array}
$$
Inequality (66) holds since  $\pi_{in}(a_t|\mathbf{s}_t)$  has a maximum value when  $a_{t} = \mu_{t,in}$ , and it shows that equation (8) is a lower bound of equation (10).
We use equation (8) instead of (10) to approximated equation (9) since it includes more information about the distribution  $\pi_{in}(\cdot |\mathbf{s}_t)$  and helps adjust the learnable variance  $\sigma_{t,in}^2$ .
A.5 Proof of Proposition 4.5
Proof. Proposition 4.4 shows that approximating  $\pi(a|\mathbf{s})$  as equation (8) comes from approximating  $\pi(\cdot|\mathbf{s})$  with
$$
\hat {\pi} (\cdot | \mathbf {s}) = \hat {w} _ {1} \mathcal {U} (\cdot ; \mu_ {1} - \gamma_ {1}, \mu_ {1} + \gamma_ {1}) + \hat {w} _ {2} \mathcal {U} (\cdot ; \mu_ {1} - \gamma_ {2}, \mu_ {1} + \gamma_ {2}). \tag {67}
$$
Let  $a$  be a continuous random variable with the probability density function being  $\hat{\pi}(\cdot|\mathbf{s})$ . Following the proof of Proposition 4.2, its first and second moments are
$$
\mathbb {E} _ {a \sim \hat {\pi}} [ a ] = \mu_ {1} \tag {68}
$$
$$
\mathbb {E} _ {a \sim \hat {\pi}} \left[ a ^ {2} \right] = \mu_ {1} ^ {2} + \hat {w} _ {1} \frac {\gamma_ {1} ^ {2}}{3} + \hat {w} _ {2} \frac {\gamma_ {2} ^ {2}}{3}. \tag {69}
$$
Equation (69) holds since the variance of  $\mathcal{U}(\cdot ;a,b)$  is  $\frac{(b - a)^2}{12}$ . Then the variance of  $a$  becomes
$$
\mathbb {V} _ {a \sim \hat {\pi}} [ a ] = \hat {w} _ {1} \frac {\gamma_ {1} ^ {2}}{3} + \hat {w} _ {2} \frac {\gamma_ {2} ^ {2}}{3} \tag {70}
$$
$$
= \frac {p}{6} \left(\hat {w} _ {1} \sigma_ {1} ^ {2} + \hat {w} _ {2} \sigma_ {2} ^ {2}\right), \tag {71}
$$
which is not related to the distance between  $\mu_{1}$  and  $\mu_{2}$ .
A.6 Learning Algorithms for KIAN
Algorithm 1: Knowledge-Grounded RL with KIAN
Input: environment  $\mathcal{E}$  with a KGMDP  $(S, A, T, G, R, \rho, \gamma)$ , where  $G = \{\pi_{g_1}, \pi_{g_2}, \dots, \pi_{g_n}\}$   
1 Initialize  $\theta$ ,  $\phi$ , and  $\mathbf{k}_e$ ,  $\forall e \in \{in, g_1, \dots, g_n\}$   
2 Observe a state  $\mathbf{s}_0 \in S$  from  $\mathcal{E}$   
3 for each time step  $t$  do  
4 // Compute weights for all knowledge policies  
5 if  $\mathcal{A}$  is a discrete action space then  
6 else if  $\mathcal{A}$  is a continuous action space then  
7  
8 Compute  $\hat{w}_{in}, \hat{w}_{g_1}, \dots, \hat{w}_{g_n}$  according to equation (1) with  $c_{t,e} = 1$ ,  $\forall e \in \{in, g_1, \dots, g_n\}$   
9 // Sample an action  
10 if  $\mathcal{A}$  is a discrete action space then  
11 Sample an action  $\mathbf{a}_t \sim \pi(\cdot | \mathbf{s}_t)$ , with  $\pi$  following equation (5)  
12 else if  $\mathcal{A}$  is a continuous action space then  
13 Sample a knowledge policy  $e \sim$  gumbelsoftmax $([\hat{w}_{t,in}, \hat{w}_{t,g_1}, \dots, \hat{w}_{t,g_n}]^\top)$   
14 Sample an action  $\mathbf{a}_t \sim \pi_{in}(\cdot | \mathbf{s}_t)$   
15 else  
16  $\mathbf{a}_t = \boldsymbol{\mu}_{e,t}$   
// Apply the action to the environment  
17 Apply  $\mathbf{a}_t \in \mathcal{A}$  to  $\mathcal{E}$  and observe a reward  $R_t$  and the next state  $\mathbf{s}_{t+1} \in S$   
// Update KIAN  
if  $\mathcal{A}$  is a discrete action space then  
19 Compute entropy-related term with  $\pi(\cdot | \mathbf{s}_t)$  following equation (5)  
20 else if  $\mathcal{A}$  is a continuous action space then  
21 Compute entropy-related term with  $\pi(\mathbf{a}_t | \mathbf{s}_t)$  following equation (8)  
22 Update  $\theta$ ,  $\phi$ , and  $\mathbf{k}_{in}, \mathbf{k}_{g_1}, \dots, \mathbf{k}_{g_n}$  with any (maximum entropy) RL algorithm  
Output:  $\theta$ ,  $\phi$ , and  $\mathbf{k}_{in}, \mathbf{k}_{g_1}, \dots, \mathbf{k}_{g_n}$
All experiments are conducted using Pytorch [22].
B.1 Baseline Algorithms
We compare KIAN with the following baselines that incorporate external knowledge policies differently.
- Behavior cloning (BC) [3]: An agent follows only the policies in  $\mathcal{G}$  to solve a task. This method is the optimal solution for supervised learning from demonstrations, where external knowledge policies generate the demonstrations.
- RL [10, 29]: An agent learns a policy by RL without any external guidance.
- RL+BC [21]: An agent learns a policy by RL with BC signals integrated. These signals come from demonstrations generated by external knowledge policies.
- KoGuN [36]: An agent learns a policy with the input being a concatenation of a state and  $n$  actions suggested by all policies in  $\mathcal{G}$ .
- A2T [27]: An agent fuses a learnable inner policy with  $n$  external policies in  $\mathcal{G}$  by learning a function approximator that predicts  $n + 1$  weights for all policies.
B.2 MiniGrid Environments
B.2.1 Environmental Details
We evaluate all methods on the following tasks in MiniGrid environments (https://github.com/maximecb/gym-minigrid): Empty-Random-5x5, Unlock, DoorKey-5x5, DoorKey-8x8, Dynamic-Obstacles-16x16, LavaCrossingS9N2, MultiRoom-N4-S5, and KeyCorridorS3R2. A state  $\mathbf{s}_t$  in each task is a directed first-person view represented as a 5x5 grid. An action  $\mathbf{a}_t$  in each task is one of the six discrete actions: left, right, forward, pickup, drop, and toggle.
B.2.2 Initial External Knowledge Set
The initial external knowledge set,  $\mathcal{G}^{init}$ , for MiniGrid tasks comprises eight sub-optimal if-else-based programs, such as:
- pick_up_the_key: If there exists a key in  $s_t$ , move to the key; if the key is in front of the agent,  $\pi_{g_j}(\text{pickup}|s_t) = 1$ .
- pick_up_the_ball: If there exists a ball in  $s_t$ , move to the key; if the key is in front of the agent,  $\pi_{g_j}(\text{pickup}|s_t) = 1$ .
- open_thedoor: If there exists a door in  $s_t$ , move to the door; if the door is in front of the agent,  $\pi_{g_j}(toggle|s_t) = 1$ .
- open_the_lockeddoor: If there exists a locked door in  $s_t$ , move to the door; if the door is in front of the agent,  $\pi_{g_j}(toggle|s_t) = 1$ .
- open_the_unlockeddoor: If there exists a unlocked door in  $s_t$ , move to the door; if the door is in front of the agent,  $\pi_{g_j}(toggle|s_t) = 1$ .
- go_to_the_goal: If there exists a goal in  $s_t$ , move to the goal.
- do_not_hit: If there exists walls or lava around the agent, do not choose the direction.
- do_not_hit_balls: If there exists balls around the agent, do not choose the direction.
In the above policies, the 'move to' is decided by  $p_{obj} - p_{self}$ , where  $\mathbf{p}_{self} \in \mathbb{R}^3$  is the position of the agent and  $\mathbf{p}_{obj} \in \mathbb{R}^3$  is the position of the object. The  $\pi_{g_j}$  of the actions right, left, and forward, can be written as:
$$
\left\{ \begin{array}{l l} \pi_ {g _ {j}} (r i g h t | s _ {t}) = 1, & \text {i f} \mathbf {p} _ {o b j, x} - \mathbf {p} _ {s e l f, x} > 0 \\ \pi_ {g _ {j}} (l e f t | s _ {t}) = 1, & \text {i f} \mathbf {p} _ {o b j, x} - \mathbf {p} _ {s e l f, x} <   0 \\ \pi_ {g _ {j}} (f o r w a r d | s _ {t}) = 1, & \text {i f} \mathbf {p} _ {o b j, y} - \mathbf {p} _ {s e l f, y} > 0 \end{array} \right. \tag {72}
$$
B.2.3 Model Architecture
The eight environments in Minigrid share the same model architecture. Each method involves learning an image encoding, an actor and a critic networks. The architecture of the image encoding network and critic network are the same for all methods, but their actor networks have different architectures.
Image Encoding Network. The image encoding network is a three-layer convolutional neural network that maps an input image to an image embedding, which is used as the state by actor and critic networks.
Critic Network. A critic network is a multi-layer perceptron (MLP) that predicts a state value [29]. The architecture of a critic network has one hidden layer that contains 64 units. Each hidden layer is followed by Tanh activation.
Actor Network of PPO, PPO+BC, and KoGuN. An actor network of PPO, PPO+BC and KoGuN is an MLP with one hidden layers and a hidden size of 64 units.
Actor Network of A2T. An actor network of A2T contains an internal actor network and an attention network. The internal actor network has the same architecture as an actor of PPO, PPO+BC, and KoGuN. The attention network is an MLP with one hidden layer and a hidden size of 64 units base.
KIAN. The internal actor network of KIAN has the same architecture as an actor of PPO, PPO+BC, and KoGuN. Each knowledge key is a learnable vector with  $d_k = 8$  and modeled by the PyTorch module, nn.Embedding. The query network is an additional output layer projecting 64-dim to  $d_k$ -dim that based on the inner actor's first layer outputs.
B.2.4 Hyperparameters
We implement all methods based on the implementation of PPO in https://github.com/lcswillems/rl-starter-files. The training timesteps are 75K, 300K, 75K, 3M, 300K, 5M, 500K, 1M for Empty-Random-5x5, Unlock, DoorKey-5x5, DoorKey-8x8, Dynamic-Obstacles-16x16, LavaCrossingS9N2, MultiRoom-N4-S5, and KeyCorridorS3R2 respectively. The learning rates are  $1 \times 10^{-3}$  for all tasks. The batch sizes are 256 for all tasks. The discount factors  $\gamma = 0.99$  for all tasks. The coefficient of the entropy term  $\alpha$  is searched to be 0 or 0.01, the default value.
B.3 OpenAI-Robotic Environments
B.3.1 Environmental Details
We evaluate all methods on the following tasks in OpenAI-Robotic environments: FetchPush, FetchSlide, and FetchPickAndPlace. A state  $\mathbf{s}_t \in \mathbb{R}^{25}$  in each task contains (1) the position and velocity of the end-effector, (2) the position, rotation, and velocity of the object, (3) the relative position between the object and the end-effector, and (4) the distance between the two grippers and their velocity. An action  $\mathbf{a}_t \in \mathbb{R}^4$  in each task contains the position variation of the end-effector and the distance between the two grippers.
B.3.2 Initial External Knowledge Set
The initial external knowledge set,  $\mathcal{G}^{init}$ , for all OpenAI-Robotic tasks comprises two sub-optimal if-else-based programs, move_forward_to_the_object and move_forward_to_the_goal.
- move_forward_to_the_object: If  $\| \mathbf{p}_{ee} - \mathbf{p}_{obj}\|_2 \geq \varepsilon$ , move straightly to the object with the gripper opened; otherwise, stay unmoved.
- move_forward_to_the_goal: If  $\| \mathbf{p}_{ee} - \mathbf{p}_{obj}\| _2 < \varepsilon$ , move straightly to the goal with the gripper closed; otherwise, stay unmoved.
In the above two policies,  $\mathbf{p}_{ee} \in \mathbb{R}^3$  is the position of the end-effector, and  $\mathbf{p}_{obj} \in \mathbb{R}^3$  is the position of the object. For all tasks,  $\varepsilon = 0.03$ .
B.3.3 Model Architecture
FetchPush, FetchSlide, and FetchPickAndPlace share the same model architecture. Each method involves learning an actor and a critic network. The architecture of the critic network is the same for all methods, but their actor networks have different architectures.
Critic Network. A critic network is a multi-layer perceptron (MLP) that predicts a state-action value  $[?]$ . The architecture of a critic network has three hidden layers, and each layer contains 512 units. Each hidden layer is followed by ReLU activation.
Actor Network of SAC, SAC+BC, and KoGuN. An actor network of SAC, SAC+BC and KoGuN is an MLP with three hidden layers and a hidden size of 512 units.
Actor Network of A2T. An actor network of A2T contains an internal actor network and an attention network. The internal actor network has the same architecture as an actor of SAC, SAC+BC, and KoGuN. The attention network is an MLP with two hidden layers and a hidden size of 64 units.
KIAN. The internal actor network of KIAN has the same architecture as an actor of SAC, SAC+BC, and KoGuN. Each knowledge key is a learnable vector with  $d_{k} = 4$  and modeled by the PyTorch module, nn. Embedding. The query network is an MLP with two hidden layers and a hidden size of 64 units.
B.3.4 Hyperparameters
We implement all methods based on the implementation of SAC in Stable-Baselines3 (SB3) [26]. The training timesteps are 1M for FetchPush and FetchPickAndPlace and 1.2M for FetchSlide. The learning rates are  $5 \times 10^{-4}$  for FetchPush and FetchPickAndPlace and  $4 \times 10^{-4}$  for FetchSlide. The batch sizes are 2048 for all tasks. The replay-buffer sizes are 1M for all tasks. The discount factors  $\gamma = 0.95$  for all tasks. The coefficient of the entropy term  $\alpha$  is adjusted automatically for all tasks as described in [10].
C Details of Compositional and Incremental Experiments
C.1 MiniGrid Environments