Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning
Authors: Zih-Yun Chiu
Yi-Lin Tuan
William Yang Wang
Michael C. Yip
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Deep Reinforcement Learning
Sample Efficiency
Generalizability
Multi-Policy Decision Making
Multi-Policy Continuous Control

Abstract: Reinforcement learning (RL) agents have long sought to approach the efficiency of human learning. Humans are great observers who can learn by aggregating external knowledge from various sources, including observations from others' policies of attempting a task. Prior studies in RL have incorporated external knowledge policies to help agents improve sample efficiency. However, it remains non-trivial to perform arbitrary combinations and replacements of those policies, an essential feature for generalization and transferability. In this work, we present Knowledge-Grounded RL (KGRL), an RL paradigm fusing multiple knowledge policies and aiming for human-like efficiency and flexibility. We propose a new actor architecture for KGRL, Knowledge-Inclusive Attention Network (KIAN), which allows free knowledge rearrangement due to embedding-based attentive action prediction. KIAN also addresses entropy imbalance, a problem arising in maximum entropy KGRL that hinders an agent from efficiently exploring the environment, through a new design of policy distributions. The experimental results demonstrate that KIAN outperforms alternative methods incorporating external knowledge policies and achieves efficient and flexible learning. Our implementation is available at https://github.com/Pascalson/KGRL.git .

Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning
Zih-Yun Chiu
Yi-Lin Tuan 2-
William Yang Wang
Michael Yip'
University of California, San Diego
University of California, Santa Barbara
Abstract
Reinforcement learning (RL) agents have long sought to approach the efficiency of human learning. Humans great observers who can learn by aggregating external knowledge from various sources, including observations from others' policies of attempting task. Prior studies in RL have incorporated external knowledge policies to help agents improve sample efficiency. However, it remains non-trivial to perform arbitrary combinations and replacements of those policies, an essential feature for generalization and transferability. In this work, we present mnowledge-Groundee RL (KGRL), an RL paradigm fusing multiple knowledge policies and aiming for human-like efficiency and flexibility. We propose a new actor architecture for KGRL, Knowledge- Inclusive Attention Network (KIAN), which allows free knowledge rearrangement due to embedding- -asted attentive action prediction. KIAN also addresses entropy imbalance, a problem arising in maximum entropy KGRL that hinders an agent from efficiently exploring the environment, through a new design of policy distributions. The experimental results demonstrate that KIAN outperforms alternative methods incorporating external knowledge policies and achieves efficient and flexible learning. Our implementation is available at littpts:: //:g::/gttttt cww/ wwon//ascaasoo/KKeie git
Introduction
Reinforcement learning (RL) has been effectively used a variety fields, including physics 7 35 and robotics [15 30). This success can be attributed RL's iterative process of interacting with the environment and learning policy to get positive feedback. Despite being influenced by the learning process of infants (32), the RL process can require a large number of samples solve task [I], indicating that the learning efficiency RL agents is still far behind that of humans.
What learning capabilities do humans possess, yet RL agents still missing? Studies in social learning have demonstrated that humans often observe the behavior of others diverse situations and utilize those strategies as external knowledge to accelerate their own exploration of solution- space. This type of learning is very flexible for humans since they can freely reuse and update the knowledge they already possess. The followings are the five properties (the last four have been mentioned 144 that summarize the efficiency and flexibility of human learning. [Knowledge- Acquirable]: Humans can develop their strategies by observing others. [Sample- Efficient: Humans require fewer interactions with the environment to solve task by learning from external knowledge. [Generalizable]: Humans can apply previously observed strategies, whether developed internally provided externally, to unseen tasks. [Compositional]: Humans can combine strategies from multiple sources to form their knowledge set. [Incremental]: Humans do not need relearn how navigate the entire knowledge set from scratch when they remove outdated strategies or add new ones.
indicates equal contribution. The corresponding emails are zchiu@ucsd, and ytuanncs.... www..uc......
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Policy
Policy
else
Joy
Joy
Observed from
Amy
Observed
Amy
Myself
from
Solved
Solved
Figure 1: An illustration of gowwledg--aqqurabbee compositional, and incremental properties in KGRL. Joy first learns to ride a motorcycle by observing Amy skateboarding and Jack biking. Then Joy learns to drive car with the knowledge set expanded by Joy's developed strategy of motorcycling.
Possessing all five learning propertics remains challenging for RL agents. Previous work has endowed an RL agent with the ability learn from external knowledge
and mitigate sample inefficiency [21 25 27 36 where the knowledge focused in this paper is state-action including pre-collected demonstrations policies. Among those methods, some have also allowed agents to combine policies in different forms to predict optimal actions (compositional)
mappings (full definition in Section
25 27 However, these approaches may not be suitable for incremental learning, in which an agent learns sequence of tasks using one expandable knowledge set. In such case, whenever the knowledge set updated by adding replacing policies, prior methods, e.g.. [27 [3), require relearning the entire multi-polit fusion process, even if the current task is similar to the previous one. This because their designs of knowledge representations are intertwined with the knowledge-fusing mechanism, which restricts changing the number of policies in the knowledge set.
To this end, our goal is enhance RL grounded external knowledge policies with more flexibility. Grounded We first introduce Knouledg-GGounded Reinforcement Learning (KGRL), an RL paradigm that seeks find optimal policy of Markov Decision Process (MDP) given set of external policies as illustrated in Figure [ We then formally define the knowledge-acquirable sample-efficient, generalizable, compositional, and incremental properties that a well-trained KGRL agent can possess.
Atlention Network (KIAN), for KGRL. KIAN consists of three components: (1) internal policy that learns self-developed strategy, (2) embeddings that represent each policy, and (3) a query that performs embedding based attentive action prediction to fuse the internal and external policies. The policy-embedddigg and query design in KIAN is crucial, as it enables the model to be incremental by unifying policy representations process. Consequently, updating adding policies to KIAN has minimal effect on its architecture and does not require retraining the entire network. Additionally, KIAN addresses the problem of entropy imbalance in KGRL, where agents tend to choose only few sub-optimal policies from the knowledge set. We provide mathematical evidence that entropy imbalance can prevent agents from exploring the environment with multiple policies. Then we introduce a new approach for modeling external-policy distributions to mitigate this issue.
We propose a simple yet effective actor model, Knowledge-Inclusee
and separating them from the policy-fusine
Through experiments on grid navigation [5] and robotic manipulation [24] tasks, KIAN outperforms alternative methods incorporating external policies terms of sample efficiency as well as the ability to do compositional and incremental learning. Furthermore, our analyses suggest that KIAN has better generalizability when applied to environments that are either simpler or more complex.
Our contributions are:
We introduce KGRL, RL paradigm studying how agents learn with external policies while being knowledge- knowledge-acquirabll. sample-efficient, generalizable, compositional, and incremental.
We propose KIAN, an actor model for KGRL that fuses multiple knowledge policies with better flexibility and addresses entropy imbalance for more efficient exploration.
We demonstrate in experiments that KIAN outperforms other methods incorporating external knowledge policies under different environmental setups.

---

2 Related Work
popular line of research in RL is improve sample efficiency with demonstrations (RL from demonstrations; RLfD). Demonstrations are examples of completing a task and are represented as state-action pairs. Previous work has leveraged demonstrations by introducing them into the policy-updatee steps of RL [I 21 23 28, 34]. For example, Nair et al. [21] adds a buffer of demonstrations to the RL framework and uses the data sampled from it to calculate a behavior- cloning loss. This loss is combined with the regular RI. loss to make the policy simultaneously imitate demonstrations and maximize the expected return. RLfD methods necessitate an adequate supply of high-quality demonstrations to achieve sample-efficient learning, which can be time-consuming In addition, they are low-level representations of a policy. Consequently, if an agent fails to extract a high-level strategy from these demonstrations, it will merely mimic the actions without acquiring generalizable policy. In contrast, our proposed KIAN enables an agent to learn with external policies of arbitrary quality and fuse them by evaluating the importance of each policy the task. Thus, the agent must understand the high-level strategies of each policy rather than only imitating its actions.
Another research direction in RL focuses on utilizing sub-optimal external policies instead of demonstrations to improve sample efficiency 25 27 (6). For instance, Zhang et al. [36] proposed Phowledge-Guidee Policy Network (KoGuN) that learns a neural network policy from fuzzy-rule controllers. The neural network concatenates a state and all actions suggested by fuzzy-rule controllers as an input and outputs refined action. While effective, this method puts restrictions representation
the policy to be a fuzzy logic network. On the other hand, Rajendran et al. [27] presented A2T (Attend, Adapt, and Transfer), an attentive deep architecture that fuses multiple policies and does not restrict the form of a policy. These policies can be non- non-primitive, and a learnable internal policy is included. In A2T, weights of all policies. The agent then samples an action from the fused distribution based on these weights. The methods KoGuN and A2T are most related to our work. Based on their success, KIAN further relaxes their requirement of retraining for incremental learning since both of them depend on the preset number of policies. Additionally, our approach mitigates the entropy imbalance issue, which can lead to inefficient exploration and was not addressed by KoGuN and A2T.
attention network takes state an input and outputs the
There exist other RL frameworks, such as hierarchical RL (HRL), that tackle tasks involving multiple policies. However, these frameworks are less closely related to our work compared to the previously mentioned methods. HRL approaches aim to decompose complex task into a hierarchy of sub-tasks and learn a sub-policy for each sub-task 6 13 16 [8] 20 methods, including KoGuN, A2T, and KIAN, aim to address task by observing a given set of external policies. These policies may offer partial solutions, be overly intricate, or even have limited relevance the task hand. Furthermore, HRL methods typically apply only one sub-policy the environment at each time step based on the high-level policy, which determines the sub-task the agent is currently addressing. In contrast, KGRL seeks to simultaneously apply multiple policies within single time step by fusing them together.
25
31 33 On the other hand, KGRL
3 Problem Formulation
Our goal is to investigate how RL can be grounded on any given set of external knowledge policies to achieve nooleege-acquirabee sample-efficient, generalizable, compositional, and incremental properties. We refer to this RL paradigm as Knowledge-Grounded Grounded Reinforcement Learning (KGRL).
A KGRL, problem is sequential decision-making problem that involves an environment, an agent, Markov
and set of external policies. It can be mathematically formulated as a Knowledge-Grounde Decision Process (KGMDP), which is defined by a tuple M₁ = AS R, p, 7. 7),, where S is the state space, AA is the action space. T S x A X RR is the transition probability distribution, R is the reward function, p is the initial state distribution, 7 is the discount factor, and G is the set of external knowledge policies. An external knowledge set 9 contains n knowledge policies, = [T ... T. ). Each knowledge policy is function that maps from the state space to the action space, N (-I-): : S - j = I,..., n. A knowledge mapping is not necessarily designed for the original Markov Decision Process (MDP), which is defined by the tuple M = (S, T, R,p.y) 7). Therefore, applying Taz to M may result in poor expected return.
3

---

The goal of KGRL is to find an optimal policy x (+1:9)= S - A that maximizes the expected Ei=0 2 Re]. Note that Mr and M share the same optimal value function, (*(s)= max ET k 7dth Ret = s), if they are provided with the same policy class II.
return: p, .
Tell
sample- efficient, generalizable, compositional, and incremental. Here we formally define these properties. Definition 3.1 (Knowledge-Acquulll Acquirable,.. An agent acquire knowledge internally instead of only following G. We refer to this internal knowledge as an inner policy and denote it as (-+(+)+) : S + A.
A well-trained KGRL agent possess the following properties:
Definition 3.2 (Sample-Efficient) An agent requires fewer samples to solve for Mr than for M. Definition 3.3 (Generalizable). A learned policy x--1:9) g) can solve similar but different tasks.
Definition 3.4 (Compositional). Assume that other agents have solved for m KGMDPs, MM. with external knowledge sets, "!.....g* and inner policies, x am.-.... "en" An agent is compositional if it can learn to solve a KGMDP M with the external knowledge set 9' SU G f Tn
Definition 3.5 (Incremental). An agent is incremental if it has the following two abilities: (1) Given a KGMDP Mg for the agent to solve within T timesteps. The agent can learn to solve Md with the external knowledge sets, .i...., GT, where Gr, f {I..... T). is the knowledge set at time step and Gr can be different from one another. (2) Given a sequence of KGMDPs M the agent can solve them with external knowledge sets, G¹ ...... .......,. where G knowledge set for task i, and G can be different from one another.
, {l....,m, m). the
4 Knowledge-Inclusive Attention Network
Environment
We propose Knowledge-Incluee Inclusive Attention Net- work (KIAN) an actor for KGRL. KIAN can be end-to-end trained with various RI. algo- rithms. Illustrated in Figure KIAN comprises three components: an inner actor, knowledge keys, and query. In this section, first de- scribe the architecture of KIAN and its action- prediction operation. Then we introduce entropy imbalance, problem that emerges maximum entropy KGRL, and propose modified policy distributions for KIAN to alleviate this issue.
action (at)
state (5t)
KIAN
TTa
hg
Meu lodge (gg)
K Know (12)
kg Knowledge (13))
4.1 Model Architecture
Inner
/b
Inner Actor The inner actor serves the same purpose as an actor in regular RL, represent- ing the inner knowledge learned by the agent through interactions with the environment. In KIAN, the inner actor, denoted as (+1: 0) S A. is learnable function approximator with parameter O. The presence of the inner actor KIAN crucial for the agent to be capable of acquiring knowledge, as it allows the agent to develop its own strategies. Therefore, even if the external knowledge policies in are unable to solve a particular task, the agent can still discover an optimal solution.
Figure 2: The model architecture of KIAN.
Knowledge Keys. In KIAN, we introduce learnable embedding vector for each knowledge policy, including win and Tg.:...: Wa.. in order to create unified representation space for all knowledge policies. Specifically, for each knowledge mapping wn or Ra; E G. we assign a learnable di-dimensional vector as its key (embedding): kin E Rd or ky, E Rd Vj E {l...., n). It is important to note that these knowledge keys, kg, represents the entire knowledge mapping me. Ve Thus, k, is independent of specific states or actions. These knowledge keys and
{in, .I........ 9n. the query will perform attention operation determine how an agent integrates all policies.
Our knowledge-key design is essential for an agent to be compositional and incremental. By unifying the representation of policies through knowledge keys, remove restrictions on the form of

---

knowledge mapping. It can be any form, such as a lookup table of state-action pairs (demonstra-
tions) 21) -f-else-based programs, fuzzy logics [6). or neural networks [25] 27 In addition, the knowledge keys are not ordered, so ag...... gg. in G and their corresponding :g...... kg. can be freely rearranged. Finally, since knowledge policy encoded as a key independent of other knowledge keys in joint embedding space, replacing policy in G means replacing knowledge key in the embedding space. This replacement requires no changes in the other part of KIAN's architecture. Therefore, agent can update G anytime without relearning significant part of KIAN.
Query. The last component in KIAN, the query, is a function approximator that generates de- fusion. The query is learnable with parameter & and is
dimensional vectors for knowledge-pollee state-dependent, so we denote it as ( Φ) : S - Rdx, Given a state 8; E S, the query outputs a d dy-dimensional vector u, = ((; +) E Rd which will be used to perform an attention operation with all knowledge keys. This operation determines the weights of policies when fusing them.
4.2 Embedding- Based Attentive Action Prediction
The way to predict an action with KIAN and a set of external knowledge policies, G, is by three steps: (I) calculating weight for each knowledge policy using embedding- based attention operation, (2) fusing knowledge policies with these weights, and (3) sampling an action from the fused policy.
Embedding- -Bsed Attention Operation. Given a state s, S, KIAN predicts a weight for each knowledge policy as how likely this policy will suggest good action. These weights can be computed by the dot product between the query and knowledge keys as:
Wt,in (s: ②) Kin /c.in E R,
Wt.9; = ((r: Φ) kg, Ct.g, E I. Vj fl.....
[iitinn Wi..g. Wt.gg. sof (ootmax(([u,inn Wh.g1 ................................ Wt lT)
where Cran E R and Ct.g; E IR are normalization factors, for example, if cr.9; = (| 1) |gg ll2. then Wt turns out to be the cosine similarity between ((s;; Φ) and kg.- We refer to this operation as an embedding-bart
attention operation since the query evaluates each knowledge key (embedding) by equation (1) to determine how much attention an agent should pay to the corresponding knowledge policy. Wr, is larger than Wr.gr the agent relies more on its self-learned knowledge policy Tini otherwise, the agent depends more on the action suggested by the knowledge policy TT Note that the computation of one weight is independent of other knowledge keys, so changing the number of knowledge policies will not affect the relation among all remaining knowledge keys.
Action Prediction for A Discrete Action Space. An MDP (or KGMDP) with a discrete action space usually involves choosing from d, E N different actions, each knowledge policy maps from a state to a d, dimmenniio probability simplex, Tin S + Ada Ty, S + A Vj = I,..., When choosing an action given state S; S, KIAN first predicts (-1ss,) A C Rd with the weights, Win; w....... "gg..
(an T T--1s) Tan (-1st) + Σリー M9 To, (1
The final action is sampled as a ~ TT- (-(ss)), where the i-th element of . (s:) represents the probability of sampling the i-th action.
Action Prediction for A Continuous Action Space. Each knowledge policy for continuous action space a probability distribution that suggests a dimdienssa action for an agent to apply to the task. As prior work (25 we model each knowledge policy as multivariate normal distribution, Tin (-(s.) [Nu,, ( Ray (+|s,) =N(H of igg Vj E (1,...,n) where Pt,in R and Ht E Rdda are the means, and of.. in. TR and a o E Rdd are the diagonals of the covariance matrices. Note that we assume each random variable in an action is independent of one another.
continuous policy fused as equation 33) becomes a mixture of normal distributions. To sample an action from this mixture of distributions without losing the important informa- tion provided by each distribution, we choose only one knowledge policy according to the weights and sample action from it. We first sample element from the set
5

---

{in, .nn) according to the weights, { Lg.g wt.g...... ......... Wr.g. }. using Gumbel softmax [1]]
gumbel_sof sumbel_softmax((() (max(( in Wt. .......... Wi g..
" ), in order to make KIAN differentiable every-
where. Then given a state S; E S, an action is sampled from the knowledge policy, a, (=(-)s)).
using the reparameterization trick.
However, fusing multiple policies as equation ( will make agent biased toward small set of knowledge policies when exploring the environment in the context of maximum entropy KGRL.
4.3 Exploration in KGRL
Maximizing entropy is a commonly used approach to encourage exploration in RL 9 T0 37 However, in maximum entropy KGRL, when the entropy of policy distributions are different from one another, it leads to the problem of entropy imbalance. Entropy imbalance is phenomenon in which an agent consistently selects only single small set of knowledge policies. We show this in math by first revisiting the formulation of maximum entropy RI. In maximum entropy RI., an entropy term is added to the standard RL objective as x argmax E, ((,. [Rss, a,) + aH (T(+)s:))
where a E R is hyperparameter, and (-) represents the entropy of distribution. By maximizing all ( (s((ss))), )), the policy becomes more uniform since the entropy of a probability distribution is maximized when it is a uniform distribution [19] With this in mind, we show that in maximum entropy KGRL, some of the weights in { { in: Wi.g...... ut.g.. might always be larger than others. We provide the proofs of all propositions in Appendix [A]
Proposition 4.1 (Entropy imbalance in discrete gpcision-makinge probability simplex E Ad is fused by { Jmh and {201,.. @m following equation (3) where π E Ad, de w 2 0 Vj E {l...., m) and Ej=1 m "y 1. If the entropy of т. is maximized and || < | oo: | < |
Assume that dg da-dimensional
I| (mmiles oloe < || 1lo, then ú² + 1.
We show in Proposition A.1 that if Tı is more uniform than Tj. then || < |T
Proposition 4.2 (Entropy imbalance in continuous control). Assume a one-dimensional policy distribution π is fused by
K 1 1 + 22 where n;- N(,,, of), 10 2 OVj E [1,2}, and 0n + u22 1.
(4)
If the variance of is maximized, and of >> of and of > (pz n2) then un 1.
We can also infer from Proposition 4.2 that the variance of π defined in equation 4 depends on the distance between μl and 112 which leads to Proposition 4.3
Proposition 4.3 (Distribution separation in continuous control). Assume one-dimensional policy distribution x is fused by equation (4). If wi, U22 of and oî are fixed, then maximizing the variance of क will increase the distance between μi and p2.
Proposition 4.1 4.2 and 4.3 indicate that in maximum entropy KGRL, (1) the agent will pay more attention to the policy with large entropy, and (2) in continuous control, an agent with a learnable internal policy will rely on this policy and separate it as far away as possible from other policies. The consistently imbalanced attention prevents the agent from exploring the environment with other policies that might provide helpful suggestions to solve the task. Furthermore, in continuous control, the distribution separation can make и perform even worse than learning without any external knowledge. The reason is that external policies, although possibly being sub-optimal for the task, might be more efficient in approaching the goal, and moving away from those policies means being less efficient when exploring the environment.
4.4 Modified Policy Distributions
Proposition 4.1 and 4.2 show that fusing multiple policies with equation ( can make KGRL agent rely on a learnable internal policy for exploration. However, the uniformity of the internal policy is often desired since it encourages exploration the state-action space that not covered by external policies. Therefore, we keep the internal policy unchanged and propose methods to modify external policy distributions in KIAN to resolve the entropy imbalance issue. We provide the detailed learning algorithm of KGRL with KIAN in Appendix A.6

---

Discrete Policy Distribution.
We modify a fusion of discrete policy distributions in equation
as
T--Is:) - in (-1se) ) + Sー= ==== Wr.n, softmax (3 Th Ta, (-1s:))
(5)
(s²)) Kin
((s)) Kg,
Wr,in (s kinl2 We.9. ( 2 kgz 112
(6)
ge = | Vj E {1....,, n.,
(7)
where Br.s, E IR is a state-and-knowledge dependent variable that scales Ty, (-|s;) change its uniformity after passing through softmax. If the value of 3t.a, decreases, the uniformity, i.e., the entropy, of coftmax([5..p og (-1s:)) increases. By introducing St.g, the entropy of knowledge policies becomes adjustable, resulting in reduced bias towards the internal policy during exploration.
Continuous Action Probability. We modify the probability of sampling a, E TR from a continu- ous T( T--(s;) in equation
π (a) = uin (nt,in + == 90 To (Mr,g (st),
(8)
where am TTnn (-1s;) and Pr.gz E Rd is the mean of TT (-|s;). We show in the next proposition that equation (8) is an approximation of
π (ar(s:) = Win (ar + -== Ta, (ar (st),
which is the exact probability of sampling a, E Rd. from a continuous ग (-|s;) in equation 3
Proposition 4.4 (Approximation of mixture of normal distributions). If the following three inequalities hold for Me.sn Pe.g...... Me.gn and atiin: || Mt ll2 < mmin minn nt.g J. ||at,in Pt.inll2 minn[ (t..;n and ||ar,in -t..glll Vt.g Vj [1,...,n). where It (2 (Ut.in|(s)) and Tt 1/(2TT (Me,o, (st)), then equation (9) for real-valued action a, sampled from KIAN can be approximated by
Wr Pt,in MMinn Pt.in + Vt,tn) + Σ Wr M(ri (e, -t,
Mt,ôn +.,95):
(10)
where U(;aa, ((];a,b)11((0 1/(bb- a).
(11)
In addition, equation 8 is lower bound of equation TT
With equation (8.. we can show that maximizing the variance of π (-((s:) will not separate the policy distributions. Hence, an agent can refer to external policies for efficient exploration and learn its own refined strategy based on them.
Proposition 4.5 (Maximized variance's independence of the distance between means). Assume one-dimensional policy x is fused by equation (4). If (als) is approximated as equation 88) and the three inequalities in Proposition 4.4are are satisfied, then maximizing the variance of x will not affect the distance between μ1 and p2-
Experiments
We evaluate KIAN on two sets of environments with discrete and continuous action spaces: Mini- Grid (5) and AppenAl-Roootic 22). Through experiments, we answer the following four questions: [Sample Efficiency] Does KIAN require fewer training samples to solve a task than other external- policy-inelusive methods? [Generalizability) Can KIAN trained on one task directly used to solve another task? [Compositional and Incremental Learning] Can KIAN combine previously learned knowledge keys and inner policies to learn a new task? After adding more external policies to G, can most of the components from a trained KIAN be reused for learning?
For comparison, we implement the following five methods as our baselines: behavior cloning (BC) (3) RL [10 29 RL+BC (21), KoGuN [36], and A2T (7). KoGuN and A2T are modified to be compositional and applicable in both discrete and continuous action spaces. Moreover, all methods (BC, RL+BC, KoGuN, A2T, and KIAN) are equipped with the same initial external knowledge set, ginit for each task. This knowledge set comprises sub-optimal if-else-based programs that cannot complete a task themselves, e.g., pickup_e a_key or move move_ frreeffowwrr___ to_ the_gtaee Ginit will be expanded with learned policies in compositional- and pncremental-leamning experiments. We provide the experimental details in Appendix B
7

---

BC
PPO
PPC+BC
PPO+KoGuN
PPO+A2T
PPO+KIAN
Empty-Randem-5x5
Unlock
Door
DoorKey-8x8
1.0
1.0
1.0
Reward
Reward
Reward
Reward
0.5
0.5
0.5
0.5
0.0
0.0
2.5
5.0
7.5
2
3
0.0
2.5
5.0
7.5
0
Timestaps
Timestegs
365
Timestops
let
Timesteps
1eG
Dynamic- Dynamicc-bstaaless Costacles-16x1
LevaCrossing59N2
MultiRoom-N4-5S
KeyCoridorS3R2
0.6
0.75
Reward
neward
Reward
Reward
0.5
0.4
0.50
0.2
0.25
0.0
0.0
0.00
Timesteps
Timesteps
166
Timesteps
105
Timesteps
BC
SAC
SAC+BC
SAC+KaGuN
SAC+A2T
SAC+KIAN
Pick-and-Place
Push
Slide
1.0
1.0
0.75
Reward
Reward
Reward
0.5
0.5
0.25
0.0
0.0
0.00
0.5
1.0
0.0
Timesteps
Led
Timesteps
led
Timesteps
Figure 3: The learning curves of sample efficiency experiments in MiniGrid (top 2 rows) and OpenAI- Robotics (last row) environments. Given knowledge set that cannot complete a task (as shown by BC), KIAN exhibits better sample efficiency across all tasks. These results underline the effectiveness of KIAN in leveraging external policies mitigate the need for extensive training samples.
Train in
Empty- Random-555
DoorKey-5x5
Push
Slide
Pick-and-Plas
Test in
6x6
8x8
16x16
8x8
16x16
5x
10x
5x
10x
5x
10x
RL [1]]29
0.88
0.71
0.45
0.29
0.08
0.87
0.52
0.45
0.17
0.34
0.27
RL+BC 21
0.87
0.60
0,24
0.40
0.09
0.89
0.60
0.44
0.16
0.34
0.30
KoGuN 36
0.94
0.83
0.53
0.77
0.35
0.63
0.43
0.55
0.18
0.32
0.24
A2T 27
0.92
0.78
0.51
0.53
0.11
0.03
0.05
0.00
0.01
0.01
0.06
KIAN (ours)
0.96
0.91
0.93
0.76
0.42
0.93
0.70
0.42
0.15
0.92
0.72
Table 1: (Zero-Shot S2C Experiments) The left five columns show the generalizability results of an agent trained in a 5x5 environment and tested in environments of varying sizes. The right six columns show the results of an agent trained with Ix goal range and tested with different goal ranges. Transferring policies from simple task to a more complex one a challenging setup in generalizability experiments. The results highlight the superior performance of KIAN in such setup.
5.1 Sample Efficiency and Generalizability
We study the sample efficiency of baselines and KIAN under the intra-task selup, where an agent learns single task with the external knowledge set gimit fixed. Figure plots the learning curves in different environments. All experiments in these figures are run with ten random seeds, and each error band is a 95% confidence interval. The results of BC show that the external knowledge policies are sub-optimal for all environments. Given sub-optimal external knowledge, only KIAN shows success in all environments. In general, improvement of KIAN over baselines is more apparent when the task is more complex, e.g.. Empty < Unlock < DoorKey and Push Pick-and-Plass Moreover, KIAN is more stable than baselines in most environments. Note that in continuous-come and Tick-and-Place) A2T barely succeeds since it does not consider the entropy imbalance issue introduced in Proposition (.2 and 4.3 These results suggest that KIAN can more efficiently explore the environment with external knowledge policies and fuse multiple policies to solve a task.
tasks (Push, Slide,
Next, we evaluate the generalizability of all methods under simple-to-complex (S2C) and complex-to- simple (C2S) setups, where the former trains policy in simple task and test it in complex one,
8

---

Train in
DoorKey-5x5
DoorKey-8x8
Pick-and-
Push Reach
Slide
Test in
Empty- -apty-
Unlock
DoorKey5x5
Reach
Push
Push
RL [10 29
0.83
0,92
0.93
0.31
0.16
0.09
RL+BC 21
0.85
0.87
0.93
0.80
0.31
0.16
0.09
KoGuN 33
0.90
0.91
0.93
0.45
0.05
0.20
0.07
A2T 2F
0.84
0.92
0.93
0.01
0.05
0.20
0.05
KIAN (ours)
0.91
0.94
0.95
1.00
0.30
0.24
0.13
Table 2: (Zero-Shot C2S Experiments) In general, KIAN outperforms other methods when transfer- ring policies across different tasks. Note that although distinguishing the levels of difficulty between Push, Slide, and Bick-and-Place is not straightforward, KIAN still achieves better performance.
KIAN (for reference)
Cont.
Cont. KoGuN
Cont. A2T
Cont. KIAN
Dynamic Dbstalll 16x16 >> LavaCrossing59N2
Dynamic @bstacles-16x1e
Dynamic Obstarles
-> MultRoom-NN4S5
> Key CorridorSRR2
Push -> Pickaand-Place
1.0
Roward
0.5
Reward
0.50
Rewardd
0.5
0.5
0.0
0.00
0.0
Timesteps
Timesteps
Timesteps
Timesteps
Unlack DoorKey-88 8x88
Deorkey-Bxxt
Door
Key CorriderSaR2
Slide
1.0
1.0
Reward
Reward
0.50
Neward
0.5
0.5
0.0
0.00
0.0
0.0
2 Timesteps
0.0
1.0
Timesteps
1es
Timesteps
Jes
Timesteps
Figure 4: The learning curves of composition and incremental experiments in MiniGrid (left 3 columns) and OpenAl-Robotics (right column) environments. KIAN requires fewer samples to learn two tasks sequentially than separately and outperforms other approaches in incremental learning.
and the latter goes the opposite way. All generalizability experiments are run with the same policies as in Section [31]] Table and show that KIAN outperforms other baselines in most experiments, and its results have a smaller variance (see Table 3 to 3Jin Appendix E). These results demonstrate that KIAN's flexibility incorporating external policies improves generalizability.
5.2 Compositional and Incremental Learning
In the final experiments, we test different methods in the compositional and incremental learning setting. We modify RL, KoGuN, and A2T to fit into this setting; details can be found in Appendix C
The experiments follow the inter-task setup: (1) We randomly select a pair of tasks (M). Mā).
(2) An agent learns a policy to solve M1 with Ginit fixed, as done in Section 5.1 (3) The learned
(internal) policy, ! is added to the external knowledge set, G ginst U (T! }. (4) The same agent
learns a policy to solve M with G. Each experiment is run with ten random seeds.
The learning curves in Figure 4 demonstrate that given the same updated G. KIAN requires fewer samples to solve M2 than RL, KoGuN, and A2T in all experiments. Our knowledge-ken key and query design disentangles policy representations from the action- prediction operation, optimized in incremental learning. Unlike our disentangled design, prior methods use a single function approximator to directly predict an action (KoGuN) or the weight of each policy (A2T) given a state. These methods make the action cettiin-rediction operation depend on the number of knowledge policies, changing the size of G requires significant retraining of the entire function approximator.
action-predictton
the agent is more
Figure also shows that KIAN solves ME more efficiently with G than ginst in most experiments. This improvement can be attributed to KIAN reusing the knowledge keys and query, which allows an agent know which policies to fuse under different scenarios. Note that & can be further expanded with the internal policy learned in M& and be used to solve another task M

---

PPO+KIAN
PRO+KIAN (Entropy Imbalance)
Empty-Random-555
Unlock
Doorkey-5x5
DoorKey-Bxa
Reward
Reward
Reward
Rewaro
0.5
0.0
0.0
2.5
5.0
7.5
2
3
0.0
2.5
5.0
7.5
2
Timesteps
104
Timesteps
343
Timesteps
Timesteps
Pynamic-bsttccies Mobssaccees-6x1t
LavaCrossingS9N2
Mult Moom-Room-4444 -4-55
KeyCorrido/3322
Reward
0.
Reward
0.5
0.5
0
2
2
0
2
0.0
0.5 Timesteps
1.0
Timestegs
Timesteps
Timesteps
SAC+KIAN
SAC+KIAN (Entropy Imbalance)
Pick-and-Place
Push
Slide
Rewerd
Revea
0.5
0.0
1.0
Timesteps
Timesteps
Timesteps
Figure 5: The learning curves of KIAN with and without addressing entropy imbalance as described in Section 4.4 The results indicate the adverse impact of entropy imbalance on KIAN's performance within the context of maximum entropy KGRL. In addition, our proposed modifications to external policy distributions are shown to be highly effective in alleviating this issue.
5.3 Analysis of Entropy Imbalance in Maximum Entropy KGRL
In ablation study, we investigate (1) the impact of entropy imbalance the performance of maximum entropy KGRL and (2) whether the proposed modifications to external policy distributions in Section 4.4 can alleviate the issue.
Figure 5 shows the learning curves comparing KIAN's performance with and without addressing the
issue. The results demonstrate that when not addressing the issue using equation (5) or (8. KIAN fails to fully capitalize on the guidance offered by external policies. We also draw two noteworthy conclusions from the figure: (1) For discrete decision- mmaking tasks, the detrimental impact of entropy imbalance becomes more evident as task complexity increases. (2) For continuous-comie tasks, entropy imbalance can degrade KIAN's performance and make it perform worse than pure RL without external policies, as shown by the results of FetchPickAndPHee FetchPush. This phenomenon can be attributed to Proposition 4.3 In contrast, by adjusting KIAN's external policy distributions using equation or (8) KGRL agent can efficiently harness external policies solve given task.
and
Conclusion and Discussion
This work introduces KGRL, an RL paradigm aiming to enhance efficient and flexible learning by harnessing external policies. We propose KIAN actor model for KGRL, which predicts action by fusing multiple policies with an embedding-basee propose modifications KIAN's policy distributions to address entropy imbalance, which hinders efficient exploration with external policies in maximum entropy KGRL. Our experimental findings demonstrate that KIAN outperforms alternative methods incorporating external policies regarding sample efficiency, generalizability, and compositional and incremental learning.
attention operation. Furthermore, we
However, it is essential to acknowledge a limitation not addressed in this work. The efficiency of KIAN, as well as other existing KGRL methods, may decrease when dealing with a large external knowledge set containing irrelevant policies. This issue is examined and discussed in Appendix F Efficiently handling extensive sets of external policies is left for future research.
Our research represents an initial step towards the overarching goal of KGRL: learning a knowledge set with a diverse range of policies. These knowledge policies can shared across various environ- ments and continuously expanded, allowing artificial agents to flexibly query and learn from them. We provide detailed discussions on the broader impact of this work and outline potential directions of future research in Appendix D
10

---

References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Reincarnating reinforeement learning: Reusing prior computation accelerate progress. Advances Neural Information Processing Systems, 35:28955-28977 2022.
[2] Pierre-Lauc Bacon, Jean Harb, and Doina Precup. AAAI Conference on Artificial Intelligence, volume 31, 2017.
The option-critic architecture. In Proceedings of the
[3] Michacl Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, 1995.
[4] A. Bandura. Social Learning Theory. Prentice-Hall series in social learning theory. Prentice Hall, 1977. ISBN 9780138167516.
[5] Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. httpa://githhb. htt.al/.wco//aaicmcc//coecaoac wtw.:e////ymmmmimiiiii... 2018.
[6] Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems, 5, 1992.
[7] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmalcki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement lcarning. Nature, 602(7897):414-419 2022.
[8] Vinicius G Goecks, Gregory M Gremillion, Vemon J Lawhern John Valasek, and Nicholas R Waytowich. Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments. arXiv preprint 1910.0991.44411. 2019.
[9] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-basse based policies. In International conference on machine learning, pages 1352 1361. PMLR, 2017.
[10] Tuomas Haamoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- maximum entropy deep reinforcement learning with stochastic actor. ICML, 2018
[11] Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Arnodd, Ian Osband, John Agapiou, et al. Learning from demonstrations for real world reinforcement learning. 2017.
[12] Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.011 2016.
[13] Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language an abstraction for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[14] Leslie Pack Kaelbling. The foundation of efficient robot learning. Science, 369(6506)9115 9166, 2020.
[15] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference Robot Learning, pages 6-1-673. PMLR, 2018.
[16] Khimya Khetarpal and Doina Precup. Learning options with interest functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9955-99555 2019.
[17] Jackyeom Kim, Scohong Park, and Gunhee Kim. Unsupervised skill discovery with hottleneck option learning. arXiv preprint arXiv:2106. 14305, 2021.
[18] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforce- ment learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[19] David JC MacKay, Information theory, inference and learning algorithums. Cambridge university press, 2003.
[20] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning, arXiv preprint arX0v:1110.01257. 2018
[21] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6292 6299. IEEE, 2018.

---

[22] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chaman, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[23] Mark Pfeiffer, Samarth Shukla, Matteo Turchetta, Cesar Cadena, Andreas Krause, Roland Siegwart, and Juan Nieto. Reinforced imitation: Sample efficient deep reinforcement learning mapless navigation by leveraging prior demonstrations. IEEE Robotics and Automation Letters, 3(4):4423 4430, 2018.
[24] Matthias Plappert, Marcin Andrychowicz. Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider. Josh Tobin, Maciek Chociej. Peter Welinder, et al. Multi-goal reinforcement learning: Chal- lenging robotics environments and request for research. arXiv preprint arXiv: 09464, 2018.
[25] Ahmed H. Qureshi, Jacob J. Johnson, Yuzhe Qin, Taylor Henderson, Byron Boots, and Michael C. Yip. Composing task-agnostic policies with deep reinforcement learning. In International Conference on Learning Representations, 2020. URL.https://ppperevvee
[26] Antonin Raffin, Ashley Hill, Adam Gleave. Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-haselines:: Reliable reinforcement learning implementations. Journal of Machine Learning .html
Research, 22(268):1-8, 2021. URL httph:t:://jml http://iwwr..rr/pppppp/////22///
[27] Janarthanan Rajendran, Aravind Lakshminarayanan, Mitesh M. Khapra, Prasanna P, and Balaraman Ravindran. Attend, adapt and transfer: Attentive deep architecture for adaptive transfer from multiple sources in the same domain. International Conference on Learning Representations, 2017. URL.
https://openrerias wwt//oorum.dd--SS5iii..
[28] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 10087, 2017.
[29] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: meei:0707764440 2017.
[30] Seungmoon Song, Eukasz Kidziński, Xue Bin Peng, Carmichael Ong, Jennifer Hicks, Sergey Levine, Christopher G Atkeson, and Scott I. Delp. Deep reinforcement learning for modeling human locomotion contral in neuromechanical simulation. Journal of neuroengineering and rehabilitation, 18(1):1-17, 2021.
[31] Martin Stolle and Doina Precup. Learning options in reinforcement learning. In International Symposium abstraction, reformuation, and approximation, pages 212-223. Springer, 2002.
[32] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[33] Wei-Cheng Tseng, Jin Siang Lin, Yao-Min Feng, and Min Sun. Toward robust long range policy transfer. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 9958 9966, 2021.
[34] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pictquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforce- ment learning on robotics problems with sparse rewards. arXiv preprint arXiv: 088777777787.. 2017.
[35] Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896) 2022.
[36] Peng Zhang, Jianye Hao, Weixum Wang. Hongyao Tang, Yi Ma, Yihai Duan, and Yan Zheng. Kogun: Accelerating deep reinforcement learning via integrating human suboptimal knowledge. In International Joint Conference on Artificial Intelligence, 2020.
[37] Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
12

---

A Proofs and Learning Algorithms of KIAN
A.1 Proof of Proposition 4.1
Proposition 4.1 (Entropy imbalance in discrete decision-making), Assume that d dimensional probability simplex πT A is fused by [TI..... ... Tm and [wr,.... (u following equation where n, A0 By 2 0 vjj E (I,..., m. and 2==1 W = 1. If the entropy of x is maximized and 1| | < || boo 1| mo.llo...................... || oo < | bo. then w1
Proef. Since E A is defined as equation
3 and its entropy is maximized,
m
(12)
do
Equation 12 holds since becomes a uniform distribution if its entropy is maximized.
Since cach 7, equation
probability simplex, it can written as
1 da
J
=
da
+
61,1,
da
+ E.,2
+ Ej.da
where
(13)
dg
E =0
Vj mL....,
(14)
1=1
Substituting equation
133
into equation
12
and, after some rearrangement, we get
t0'
11.1.
Σ
m
Wi
1
+
E111
+ Ejida
(15)
da
du
da
da
do
2=
Without loss of generality, we can assume that
max E1, E1,1 c
(16)
)
min <E( (c.......
and
Eg1 50 0 Vj m2.
(17)
Then take the infinity norm for both sides of equation
15
we get
1
+ E1,1
do
+ di
d
d.
i
E
d.
Ejda
j=2
(18)
1
W1
+
Σ y=2
00
+
(19)
dg
do
After some rearrangement, equation
19
becomes
1
m
m
+ 11E1111
(20)
d
d,
)=2
121
11
Σ
(21)
2=2
Equation 21 holds since E5 08
Given equation 14 and the assumption that ||TT|//0 & | & [Tsall....., || we have the following incqualitics
& -Ei
Vj (L....,m).
(22)
Hence, for equation
21 to hold, di +1 1.
Proposition 4.1 states that if || is much smaller than || loc, then maximizing the entropy of a results in 1i + and W 0, Vj (2...., m.}. Next, we provide another proposition showing that if क uniform distribution and ny is not, then || < |o= Vj E (2...., ..).
Proposition A.1 (Infinity norm of probability simplex). Given two d. di-demessoa probability simplices, dd and E dd, if n1 is uniform distribution and wı is not, then || < T|
13

---

Prouf.
Since E A is uniform distribution,
-Ila dta
I.
Ta
The infinity norm of x becomes
|Ti |00
max (E(......)) d..
dea
(23)
where ,i, is the i-th element of mi On the other hand, since E Ad is not uniform distribution, can be represented as
M2
62,1; dg + 62,20...1 da 82.da
where
(24)
da
Σ 22 = and {52,1,.... 52 1:2, * 0} # V.
(2.5)
Equation
25
indicates that at least one element in {E2,1..... (2,de) should be larger than O. Hence,
|2/
max se (a)
[22
max se(l
+ E2.1
> da
(26)
do
A.2 Proof of Proposition 4.2
In continuous action space, the final policy fused by equation B is mixture of normal distributions, also known Gaussian mixture. However, in general, the entropy of Gaussian mixture does have closed form 45] Instead of analyzing the entropy æ,, we analyze the variance for de in maximum entropy KGRL since for any prohability density function of a real-valued random variable, Shannon's 's inequality for entropy and
variance 44 47) specifies
H(T) - 1 In (2po²)
(27)
where o² E Rgo is the variance of and 3.14159.
Proposition 4.2 (Entropy imbalance in continuous control), Assume one-dimensional policy distribution fused by
Wi + d2222, where n, = NM1,, g uy 0.j E (1,2), and th + t0 1. If the variance of π is maximized, and of >> of and of >> (μ4 n2) ), then ii + 1.
(28)
Proof. Let be continuous random variable with the probability density function being (--ss) Then its first and second moments are
/ an (x(s)s)
Easa [a)
(29)
/ a (1i1m(aas)) + )2 da
(30)
un fami(als) da + u2 aTa(aas) da
(31)
(32)
Eawo [o2]
a² (-Is)
(33)
/ 3 (1i1T (als) + ((lzs) da
(34)
in / am (als) da dz / d2 da
(35)
w6 (uì f w2(ي +z))
(36)
Equation
36 holds since 0j = Eay~a,,(0 [a (Bny (a,1) 2 .Vj [1,2), variance a thus becomes
Vana [a] = Eanx [a"] (Ba~a (a))²
(37)
ri (Hr 01) uig ( + oz) - (1 + 222
(38)
ūi ū20 in (1 ún }/² + wz(1 w2) 2m0 02/h1
(39)
r +
3 211122
(40)
áooe 620 22)²
(41)
According equation 41 of o3 and of > (w) p2)2 maximizing Vawr(a] leads in +
L
14

---

A.3 Proof of Proposition 4.3
Proposition 4.3 (Distribution separation in continuous control). Assume mee-iimnnsiona policy distribution is fused by equation 4. If úi, W2, of and of are fixed, then maximizing variance of will increase distance between μi and p2.
Proef. According to equation (41) if ih, d2, of, and ož are fixed, maximizing Vana [a] results in maximizing (#) pz)² hence increasing the distance between μai and jiz.
A.4 Proof of Proposition 4.4
Before proving Proposition we first show that KL divergence between a mixture of uniform distributions and a mixture of normal distributions is aoper-bounded by constant.
Proposition A.2 (KL divergence between mixture of uniform distributions and Gaussian mixture). Given Gaussian mixture
x(-) =
Σ wy (13
(42)
where N(x og)
1
exp -(x (xx- mb))
(43)
2200
20²
and a mixture of m uniform distributions
*()) Σ ús u((5m) - 13.44 75),
(44)
where u(: a. b a and v 2 ((pssm)).))
(45)
for real-valued random variable, the KI. divergence between (()) and (() has an upper bound of fh
Proof. Since KL divergence (KI(PIQ) is convex in the pair (P, Q), DKI ([r) has the following upper
hound 43
Dxz(fl|m) = DEL Σ by U(ius - 79.44 + 75) Σ ús N(31s
(46)
úy DEL (M(:s)) - 73.19 +35) ((:HMssf))))
(47)
For each j mm..
DKL (U(=)s Ms + +7) [[:(s)ss)))) of))
(48)
27,
In
dx
(49)
pg-
21b
(m- 203
exp
|m
1. In
1
1
1
(x
dx
In
dx
(50)
pg Py-7
2nd
27)
-3- pg
27
2po
20²
(w ta
In 27g
In
dn
(51)
/ /2p03
17307
23
In
In
(52)
275
/ 2po
6o
(53)
12'
52 Equation 53 comes from substituting " into equation
15

---

Finally, the upper bound of ( becomes
DKI(f((T)
ū DEL (U(:11- -1: + ((:;u,,0))))))
(54)
jm1
m
p
p
(55)
12
12
==1
Proposition 4.4 (Approximation a mixture of normal distributions). If the following three inequalities hold for pe,an (1..1..... Ht.... and ar,in? I| Mt.a9llz la < mnn No.93 ). min (i,,in Te.3 ). and ||at,in He.gg ll2 < n.g, Vj E {l....,niy n.). where n, = 1/ (pt. [Se]] and "n. = 12 (Wr.a, [Se]], then equation for real-valued action sampled from KIAN can be approxi- mated by
10 pr.in l| <
ir,inll(at; js,in "yiin Meiin + (n
Wy Maae Meiin "hag Maiin n.ash
(56)
sml
where u(;aab)) 1/(b a).
(57)
In addition, equation (8) is a lower bound of equation
10
Proof. Proposition A.2 shows that T(-(s,) fused as equation
can be approximated by
*(-[s:) = Wr. Ta,n Jran + Tt.sn)
Σ (( - Thoss He.o; + 7.,o))
(58)
with KL divergence being most 12 which is constant.
Since any continuous action a, outputted by KIAN belongs to {ar,in 11.01 ... pegn) (Line 11 to 16
Algorithm if three inequalities the proposition statement hold, then for all as
Ulai pa.i Hiin Meiin Tb,in)
(59)
27e.in 1
U(as p1.g5
"M,gg M4.95 + ()as))
n). and Vj E fl....,,
(60)
27/1.83
U(as pr.n "eay Me,in + Th.9))
Vj E nI.
(61)
24y..5
Therefore, (t(sss) can be rewritten as
#(- (c) (e.enld(; Me Messn Pa.amyMe,aa + (e,n)
Σ
+ -.,))
(62)
f=1
For any {at,in Mt. H....
A (ar)s:) = Wi, peai
"M Me,in + )n +
Wr M(ariMe,5n -M.gs Me,on + Th,s)
(63)
Wt 2
We.9; 271.99
(64)
1=1
-i..n Ten (Me, (sc) +
We. Tay (Me.es (S:)
(65)
dr (ar (sc)
de..g Fuy (pi.e, (st).
(66)
#=1
Inequality 666 holds since Ten (oz (s:) has maximum value when o, fat.n and shows that equation is
a lower bound of equation (T0).
We use equation 8 instead of 10 to approximated equation 9 since it includes more information about the distribution (- and helps adjust the learnable variance di.in-
16

---

A.5 Proof of Proposition 4.5
Proposition 4.5 (Maximized variance's independence of the distance between means). Assume one- dimensional policy is fused by equation 4.. If =(als) is approximated as equation (8) and the three
inequalities in Proposition 4.4 are satisfied, then maximizing variance of =+|ss) will not offect the distance
between μi and u2z-
Proof. Proposition 4.4 shows that approximating (a]s) as equation comes from approximating at-ss) with
*(-ss) o., 71, /41 71) ( 72, (22)-
(67)
Let be continuous random variable with the probability density function being #(-(s). Following the proof of Proposition 4.2 its first and second moments are
Bana a) = 1M1
(68)
22
E Eawa iu?] 11
+
in
71
+
122
(69)
3
3
Equation
holds since variance of (f
(n-o)²
Then the variance a becomes
12
71
12
V [a]
(70)
3
3
(wiof of 2222))
(71)
which is not related to distance hetween fa and f/2.
A.6 Learning Algorithms for KIAN
Algorithm 1: Sewpwlldge-rroundee RL with KIAN
Input: environment & with KGMDP A, G,R.P, where G {T Tg2 ................................ agn {
Initialize 0, Φ. and k., Ve E [in, (in,9i......
Observe state Sy E from for each time step t do
// Compute weights for all knowledge policies if A is discrete action space then
Compute Win; Wg..... Wg,, according to equation (6) else if A is a continuous action space then
Compute Win, Wgg..... Wy, according to equation with Ct.e 1, Ve E {in, .1.....9... Compute Win, ......., Ww, according to (2)
// Sample an action
if A is discrete action space then Sample an action a x(-(s),, with following equation (5)
10
11
else if A is a continuous action space then
Sample knowledge policy ~ gumbel_se soff tmax [w)t, 0r. Wt.g.. IT)
12
if in then
14
Sample an action a, ~ (-(()))))
15
else
16
a: Me,
// Apply the action to the environment Apply a, E & and observe reward R, and the next state St+1 E // Update KIAN
17
18
discrete action space then Compute entropy-related term with r-|s:;) following equation a continuous action space then
19
5
20
else
21
Compute entropy-relate term with TT (ayy)s)) following equation 8
22
Update 0, Φb and kin kg....., kg. with any (maximum entropy) RL algorithm
Output: 0, Φb and kin. kg....., kg.
17

---

B Experimental Details
All experiments are conducted using Pytorch [22]
B.1 Baseline Algorithms
We compare KIAN with the following baselines that incorporate external knowledge policies differently.
Behavior cloning (BC) A An agent follows only the policies in & to solve a task. This method is the optimal solution for supervised learning from demonstrations, where external knowledge policies generate the demonstrations.
RL T0
An agent learns a policy by RL without any external guidance.
RL+BC 21 An agent learns policy RL with BC signals integrated. These signals come from
demonstrations generated by external knowledge policies.
KoGuN [36]: An agent learns policy with the input being a concatenation of a state and actions suggested by all policies in G.
A2T (27]: agent fuses a learnable inner policy with external policies by learning function approximator that predicts n + weights for all policies.
B.2 MiniGrid Environments
B.2.1
Environmental Details
We evaluate all methods on the following tasks in MiniGrid environments /https://github. h gym- minigriddd:::
Unlock, DoorKey-5x5, DoorKey-8x6, Dynamic-Obstaoles-- 16x 16, LavaCrossingSNN2, WoltiRoom-N44SS and KeyCorridorS3R2. A state in each task is a directed first- person view represented as 5x5 grid. An action in each task is one of the six discrete actions: left, right, forward, pickup, drop, and toggle.
B.2.2 Initial External Knowledge Set
The initial external knowledge gimit, for MiniGrid tasks comprises eight sub-optimal if-clse-basee based programs, such as:
pick_up_tii key: If there exists key in st, move to the key; if the key in front of the agent,
Ty (pickupls.) sx) =
pick_up_the ball: If there exists ball in 8t, the key; if the key is in front of the agent,
T (pickupls:) sx) =
open_the, _oorr If there exists door in move to the door; if the door is in front of the agent, Tyz (toggle)sz)
cch_locked____cc If there exists a locked door in st, move to the door; the door is in front
of the agent, ø, (toggle(s:) =
there exists a unlocked in st, move to door; if the door front the agent, Taz (foggle|ss) =
open_the unlocked_door
go_to_the__goe If there exists goal in 8t, move to the goal.
do_not_hit: If there exists walls lava around agent, do not choose the direction.
do_not_hit_baat If there exists balls around the agent, do not choose the direction.
In the above policies, the 'move to' is decided by Potj Pactf, where Panty R is the position of the agent Paby Iga the position of object. The mgz of the actions right, left, forward, can be written as:
Tgy (right)sz) = 1, if Pobj.a Dself.a >
Rg, (lef([s:) = L, if Pobj,a Padlfe <
(72)
Raz (forward(ss) 1,
if Pobiay Doelf.y >
B.2.3 Model Architecture
The eight environments in Minigrid share the same model architecture. Each method involves learning an image encoding, an actor and critic networks. The architecture of the image encoding network and critic network are the same for all methods, but their actor networks have different architectures.
18

---

Image Encoding Network. The image encoding network is three-layer convolutional neural network that maps input image image embedding, which used the state by actor and critic networks.
Critic Network. critic network multi-layer pereeptron (MI.P) that predicts value 29 The architecture a critic network has one hidden layer that contains 64 units. Each hidden layer followed by
Tanh activation.
Actor Network of PPO, PPO+BC, and KoGuN. An actor network of PPO, PPO+BC and KoGuN is an MI.P with one hidden layers and hidden size of 64 units.
Actor Network of A2T. An actor network of A2T contains internal actor network and attention network. The internal actor network has same architecture as actor PPO, PPO+BC, and KoGuN. The attention network an MI.P with one hidden layer and a hidden size of units base.
The internal actor network of KIAN has the same architecture as an actor of PPO, PPO+BC, and KoGuN, Each knowledge key is a learnable vector with dN = 8 and modeled by the Py module, nn Embedding. The query network is additional output layer projecting 64-dim da-dim that based on the inner actor's first layer outputs.
KIAN.
B.2.4 Hyperparameters
implement methods based on implementation of PPO in httpps://githbb comm:/si//o//csilemmmme...
rl-starter-fill files The training timesteps are 75K, 300K, 75K. 3M. 300K. 5M, 500K, 1M for Rappo---and Unlock. DoorKey-5x5, DoorKey-8x88 Dynamic-Obstiiieee Obstacles-16x166 LavaCrossing9NNZ MultiRoom-NN N4-S5, and 3RyCoridorSS322 respectively. The lcarning rates 1 10 for all tasks. The batch sizes are 256 for all tasks, The discount factors y 0.99 for all tasks. The cocfficient of the entropy term searched be or 0.01, the default value.
B.3 OpenAI-Robotic Environments
B.3.1
Environmental Details
evaluate methods on the following tasks in OpenAl-Robotic environments: FetchPush, FetchSlide, and FetchPickAndPlace. A state S; E [8 in each task contains (1) the position and velocity of the end-effector, (2) the position, rotation, and velocity of the object, (3) the relative position between the object and the addeffector, and (4) the distance between the grippers and their velocity. An action R in each task contains the position variation of the end-effector and the distance between the two grippers.
B.3.2 Initial External Knowledge Set
imitial external knowledge set, givit, for OpenAl-Robotic tasks comprises two sub-optimal if-else-based programs, hove__orwwrd__o_thee tbojjetteeeeetetttteeeeeeeeeeeee and move_forwarde to_ to_the_goal.
Love_forward_cee theeettttt____ettteeeeeeeeeeeeee sccchhoj_ettttee If |per Possl12 2 move straightly to the object with the
gripper opened; otherwise, stay unmoved.
move_forward_to to_ttt_ttttteooooooo toall If ||eee Paty ll2 < move straightly to the goal with the gripper
closed; otherwise, stay unmoved.
above two policies, Pee E R the position of the end-effector, and Pasj E B is the position object. For all tasks, 0.03.
B.3.3 Model Architecture
FetchPush, FetchSlide, and AnddPPickAndPPace share the same model architecture. Fach method involves learning actor and critic network. The architecture of the critic network is the same for all methods, but their actor networks have different architectures.
Critic Network. A critic network is multi-layer perceptron (MIP) that predicts state-action value [? ]. architecture of critic network has three hidden layers, and each layer contains 512 units. Each hidden layer followed by ReLU activation.
Actor Network of SAC, SAC+BC, and KoGuN. An actor network of SAC, SAC+BC and KoGuN is an MLP with three hidden layers and hidden size of 512 units.
19

---

Actor Network of A2T. An actor network of A2T contains internal actor network and an attention network. The internal actor network has the same architecture as actor of SAC, SAC+BC, and KoGuN. The attention network an MLP with two hidden layers and a hidden size 64 units.
The internal actor network of KIAN has the same architecture as an actor of SAC, SAC+BC. and KoGuN. Each knowledge key is learnable vector with d2 = 4 and modeled by the PyTorch module. nn. Embedding. The query network is an MLP with two hidden layers and hidden size of 64 units.
KIAN.
B.3.4 Hyperparameters
We implement all methods based on the implementation SAC in Babbl--Bs (SB3) 26]. The training timesteps are IM for FetchPush and FerchPickAndPlace and 1.2M for FetchSlide. The learning rates are 5 x 10 for FetchPush and FetchPickAndPlan and 4 x 10 for FetchSlide, The batch sizes are 2048 for all tasks. The replay-buffe sizes are IM for all tasks. The discount factors Y 0.95 for all tasks. The coefficient of the entropy term is adjusted automatically for all tasks as described in [0
C Details of Compositional and Incremental Experiments
C.1 MiniGrid Environments
After learning an actor in MM with the experimental setup described in Section B train KIAN for MI by initializing its external knowledge keys with the knowledge keys learned Mj. These external knowledge keys remain fixed when learning ME. and all other components of KIAN are learned from scratch. This setup allows us to test the efficacy of reusing learned knowledge keys across different tasks. All actor components of RI, KoGuN, and A2T are learned from scratch in M This is because RI does not incorporate any external knowledge, and the model architectures knowledge policies. The hyperparameters of learning M, and M2 are the same as those listed in Section B.2.4 The external knowledge policies used for each experiment are detailed as follows:
KoGuN and A2T do not allow changing the number and order of
Dynamic-Ohstacles 16x16 + Srosinnnn Reuse the knowledge embedding of "go to the goal"; reuse the knowledge embedding of "do not hit balls" as the fixed knowledge embedding of "do not hit".
Unlock + DoorKey-8x8. Reuse the knowledge embeddings of "get the key" and "open the door".
Dynamic-Obstaaless-155e MultiRoom-N4-S5, Reuse the knowledge embedding of "go to the goal".
Door 8x8 + MultRooom-N4-SS. Reuse the knowledge embedding of "go to the goal"; reuse the knowledge embedding of "open the door" as the fixed knowledge embedding of "open the unlocked door".
Crramic-Osstacles-666 - KeyCorridor33R2. Reuse the knowledge embedding of "go to the goal" the fixed knowledge embedding of "pick up the ball".
DoorKey-8ke + KeyCorridorS3R2. Reuse the knowledge embedding of "get the key"; reuse the knowledge embedding of "open the door"; reuse the knowledge embedding of "go the goal" as the fixed knowledge embedding of "pick up the ball".
C.2 OpenAI-Robotic Environments
After learning an actor and a critic in MM with the experimental setup described in Section we initialize the networks for M as follows:
RL: The actor and critic of M are initialized with that of M. These networks will be updated when learning M2.
KoGuN: Only the critic of M is initialized with that of M!. The actor learned from scratch in M The actor and critic will be updated when learning in Mt.
A2T: The critic of Mg initialized with that of M.. The inner policy and attention network are learned from scratch in MG. All networks will be updated when learning in M
The external knowledge keys, query, and critic of M are initialized with that of M Note that the external knowledge keys of MM include the internal and external knowledge feysfrom from Mi. The inner policy and inner knowledge key are learned from scratch in M The external knowledge keys remain fixed, while other networks will be updated when learning in M?.
KIAN:
20

---

The hyperparameters of learning M are the same as those listed in Section B.3.4 When learning M2. the hyperparameterss changed are listed follows: The training timesteps 0.4M. The learning rates 7.5 x and 10 10-a for FetchPush FeichPickAndPlace respectively.
D Broader Impact and Future Research Directions
The KGRI. framework presented in this work aims to enhance agent's ability learn from external policies. These policies encompass only sub-optimal strategies task completion but also regulative policies that emphasize safety constraints ethical behaviors. Being able to incorporate safety- and etlhics-oriented policies gives KGRL the potential to significantly influence an artificial agent's behavior, promoting enhanced safety and social acceptabilityy. These aspects have gained substantial attention in the field of RL 38
46 48
underscoring their importance in contemporary rescarch.
Moving forward, there several research directions in KGRL that worth exploring. First, fusing knowledge policies with different state and action spaces enables efficient learning Second, integrating regulative policies that enforce strict constraints during learning and inference stages can ensure adherence to safety and ethical considerations. Lastly, addressing complex relationships among different policies, such as conditional dependence and confficts, allows an agent to efficiently navigate through large and diverse set of knowledge policies. We hope these directions have the potential to inspire future studies in KGRL.
broader range of applications.
E
Other Experimental Results
We provide the standard deviation for the experiments of generalizability in Table
Train
Emptt-Random-s
D50r
Test
6x6
8x8
16x16
8x8
16x16
10
088
271
35
.29 16
.08 10
RL+BC 21
.87
160
17
.0
.09
KoGuN
36
94
831.
.31.11
.771.09
.35
A2T
27
.92
.78
.51..00
.53+.09
.11.0
KIAN (ours)
.66+.22
.91..1
.93+.
.6+
.42+.08
Table 3: MiniGrid Zero-Shot S2C Experiments.
Train in
Push
Slide
Bick-and-Place
Test in
5x
10x
5x
10x
5x
10x
RL 10
29
.87..05
.2...11
.4.
.172.05
34
.271.44
RL+BC 21
88
.60
14
.162.02
.34..5
.30.500
KoGuN
.33
.43
.55
.182.04
.32..52
.24 38
A2T 20
.03 00
.05
.00
.01
.01
.062:00 00
KTAN (ours)
.93
.70
142
.15
.92..00
.72
Table 4: OpenAI-Robotics Zero-Shot S2C Experiments.
Train
DeorKey-5x5 Empty-Rane Random
DoorKey-8x8 DoorKey5x5
Pick-and- -lacee
Push Reach
Slide
Test in
Unlock
Reach
Push
Push
RL (10 29
834
.92
.93+
.0
.312.19
.662.10
.09
RL+BC 21
.8 05
87
93:..1
80
.312.19
16
.09.00
KoGuN 36
90 02
911
.3.1
.45
.05
.20
.07
A2T 27
884
.92+
.33+.00
.01
.05
.200..
.05
KIAN (ours)
.91
.94
.95+.00
1.0=
.30
.24
.13
Table 5: Zero-Shot C2S Experiments.
Effects of Extensive External Knowledge Set with Irrelevant Policies
In order for KGRL agents to effectively leverage an external knowledge set, it is imperative that they can (1)
distinguish which external policies less related to the given task and (2) efficiently navigate through an
21

---

extensive collection of external policies. Failure accomplish these objectives in timely manner could result in suboptimal performance, potentially inferior to that of RL agent. Under such cincumstances, integrating external policies the learning process becomes impractical.
this section, examine the following two aspects of KIAN: (1) the ability of KIAN to rapidly acquire valuable strategies, even in the presence of random irrelevant policies within the external knowledge set, and (2) the impact of KIAN's performance the size of external knowledge set increases.
PPO
PPO +PO+KIAN
PPO-KIAN--RedundanniX
Figure shows the learning curves PPO and PPO+KIAN, considering various numbers of external policies: 2 relevant, 4 (2 relevant + 2 irrelevant). and 6 (2 relevant 4 irrelevant) for the MiniGrid Unlock task. The results indicate that including vant knowledge policies leads to a marginal decline in performance, but the agents consistently achieve high rewards with minimal variances. This minor decline performance aligns with expectations since the agents need to (I) navigate through more extensive set of external policies and (2) distinguish and dis- regard policies that do not contribute to solving the task. Therefore, when the external knowledge set is very large, KGRL methods, such KoGuN, A2T, and KIAN, not guarantee superior efficiency over RI. methods. Efficiently harnessing a large external policy set remains an avenue of future research and exploration.
Unlock
1.0
Reward
irrele-
0.5
0.0
2
3 1e5
Timesteps
Figure 6: Learning curves of PPO and PPO+KIAN for the MiniGrid Unlock task, with external knowl- edge sets that include irrelevant policies.
22

---

Reference
[38] Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement lcarning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3387-3395, 2019.
[39] Yinlam Chow, Ofir Nachum, Edgar Duenez- Guzman, Mohammad Ghavamzadeh. Iyapunov approach to reinforcement learning. Advances in neural information processing systems, 31, 2018.
wvapunov-based based
[40] Dongsheng Ding, Xiaohan Wei, Zhuoran Yang. Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dus -uall policy optimization. In International Conference on Artificial Intelligence and Statistics, pages 3304-3312 PMLR, 2021.
[41] Adrien Ecoffet and Joel Lehman. Reinforeement learning under moral uncertainty. International conference on machine learning, pages 2926-2936. PMLR, 2021.
[42] Javier Garcia and Fernando Feráández. Safe exploration of state and action spaces reinforcement learning. Journal of Artificial Intelligence Research, 45:515 45:515-564, 2012.
[43] John R Hershey and Peder Olsen. Approximating the kullback leibler divergence between gaussian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing- ICA volume 4, pages IV-317. IEEE, 2007.
[44] Isidore Hirschman. A note on entropy. American journal of mathematics, 79(1):152-155) 156, 1957.
[45] Marco F Huber, Tim Bailey, Hugh Durrant- Whyte, and Uwe Hanebeck. On entropy approximation for gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pages 181-188. IEEE, 2008
[46] Puze Liu, Davide Tateo, Haitham Bou Ammar, and Jan Peters. Robot reinforcement learning the constraint manifold. In Conference on Robot Learning, pages 1357-1366. PMLR, 2022.
[47] Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3): 379 423, 1948.
[48] Yueh-Hua Wu and Shou-De Lin. A low-cost ethics shaping approach for designing reinforcement learning agents, In Proceedings the AAAI Conference on Artificial Intelligence, volume 32, 2018.
23


Reviews:
[{"review_id": "08nhTdQjGR", "reviewer": "Reviewer_7i4w", "summary": "This paper defines the Knowledge-Grounded RL setting, a general RL setting for integrating knowledge (in the form of policies) into a policy to learn new tasks efficiently. Essentially this setting is similar to the MDP setting except that the agent is also given a set of knowledge policies to utilize. The paper also introduces a system/architecture within this KGRL setting, called Knowledge-Inclusive Attention Network (KIAN). The aim is to improve RL that is grounded on external knowledge policies. The paper outlines five desirable human-like properties they desire in their agents: knowledge-acquirable, sample-efficient, generalizable, compositional, and incremental. Moreover, they formally define these so that they are measurable within the KGRL setting (e.g., for evaluating algorithms on these dimensions).\n\nWhile previous methods typically intertwine knowledge representation and knowledge-fusion, thereby restricting their ability to adapt to numbers of policies, losing flexibility. KIAN is developed more flexibility, separating the knowledge representation and knowledge fusion.\n\nKIAN consists of three components: a policy that learns a strategy (similar to a normal policy) called the internal, embeddings that represent the given knowledge (or external) policies, a query that performs attentive action prediction to fuse the internal and external policies.\n\nKIAN also solves other issues that can occur in entropy-regularized KGRL. Entropy-regularized RL is common, but the authors show that in the KGRL setting issues can arise through entropy regularization where only a select few policies are selected, counterproductively reducing diversity in policy usage. The authors show that in the KGRL setting the agent will pay more attention to the policy with large entropy and in continuous control, will rely on the internal policy extensively. The paper introduces modifications so that this does not occur in KIAN.\n\nThe authors show results on both MiniGrid and robotics tasks and demonstrate sample efficiency, generalizability, as well as compositional and incremental learning.\n", "questions": "In line 33, it is stated (for incremental): “Humans do not need to relearn how to navigate the entire knowledge set from scratch when they remove outdated strategies or add new ones”. Do humans truly “remove” outdated strategies?\n\nIn line 222, the paper states: “However, fusing multiple policies as equation (3) will make an agent biased toward a small set of knowledge policies when exploring the environment.” I am somewhat confused, equation (3) as is does not seem to have this issue. I thought this only occurs in the MaxEnt KGRL setting as introduced in the next section. Can the authors please clarify this?\n\nThe authors state the entropy imbalance problem is a property of the maximum entropy KGRL setting. I want to clarify that this is only shown for the specific case of policies in the form of equation (3), correct?\n\nHow could the policy embeddings/knowledge keys be learned?\n\nIs Gymnasium Robotics used for the experiments? Or the older OpenAI codebase?\n", "limitations": "To me it is unclear how in more sophisticated settings how the knowledge keys (or policy embeddings) would be learned. This seems like a bottleneck to scalability that is not well-addressed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is mostly well-written and well-explained.\n\nThe method makes sense, and is a well-thought out architecture. \n\nI like how the authors address the entropy imbalance problem.\n\nI like that the authors define and quantify the behaviors they would like in the agent.\n\nThe results do seem to demonstrate their method is effective.\n", "weaknesses": "While I think the experiments are good, with appropriate baselines and good environments to test the agent’s capabilities. I am concerned about statistical significance. In particular, only 5 seeds are run, and the performance benefit in many cases is minimal, which may quite possibly be attributed to randomness. While I do believe the method outperforms the baselines, I cannot say so with a lot of confidence to merit inclusion at NeurIPS. If the authors can run more seeds, especially given the large variance, it would dramatically improve their results.\n\n\nQualms:\nKGRL is consistently referred to as an RL framework, which it is, but the connotation can be misconstrued as being a framework “for” RL, implying it is a solution method for RL problems. I would recommend calling it a “setting” rather than a framework. Indeed, I was confused temporarily as a reader, especially when KGRL is stated as being “introduced” by this paper (as opposed to “described” or “outlined”). \n\n\nNits\nTypo Line 345: “border” should be: “broader”\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. *I am concerned about statistical significance. In particular, only 5 seeds are run, and the performance benefit in many cases is minimal, which may quite possibly be attributed to randomness.*\n\nThank you for pointing this out! We have increased the number of random seeds run for all experiments. Please refer to “General Response - Experimental Significance”.\n\n2. *Qualms: KGRL is consistently referred to as an RL framework, which it is, but the connotation can be misconstrued as being a framework “for” RL, implying it is a solution method for RL problems. I would recommend calling it a “setting” rather than a framework. Indeed, I was confused temporarily as a reader, especially when KGRL is stated as being “introduced” by this paper (as opposed to “described” or “outlined”).*\n\nThank you for the suggestion! We agree that “setting” is a better term for this work and will update our final paper accordingly!\n\n3. *Nits Typo Line 345: “border” should be: “broader”*\n\nThank you for pointing out the typo! We will correct it in the paper :)\n\n4. *In line 33, it is stated (for incremental): “Humans do not need to relearn how to navigate the entire knowledge set from scratch when they remove outdated strategies or add new ones”. Do humans truly “remove” outdated strategies?*\n\nWe believe that when encountering a task, humans may collect a knowledge-policy set from a large pool of policies that they think might be useful for the task. If we find a policy in this smaller knowledge set that is no longer helpful for solving the task, we might remove it from the set.\n\n5. *In line 222, the paper states: “However, fusing multiple policies as equation (3) will make an agent biased toward a small set of knowledge policies when exploring the environment.” I am somewhat confused, equation (3) as is does not seem to have this issue. I thought this only occurs in the MaxEnt KGRL setting as introduced in the next section. Can the authors please clarify this?\nThe authors state the entropy imbalance problem is a property of the maximum entropy KGRL setting. I want to clarify that this is only shown for the specific case of policies in the form of equation (3), correct?*\n\nYes, you are right! Fusing multiple policies with equation (3) will make an agent biased toward a small set of knowledge policies when exploring the environment by maximizing the entire policy’s entropy. We will clarify this point in the final paper.\n\n6. *How could the policy embeddings/knowledge keys be learned?*\n\nThey are jointly (end-to-end) learned with other components in KIAN. The gradient signals calculated by an RL algorithm, e.g., policy gradient, will be back-propagated to all policy embeddings. We provided the complete learning algorithm in Appendix A.6.\n\n7. *Is Gymnasium Robotics used for the experiments? Or the older OpenAI codebase?*\n\nWe use the older OpenAI codebase. The version we use lies within the Python package “gym” of version 0.21.0.\n\n8. *To me it is unclear how in more sophisticated settings how the knowledge keys (or policy embeddings) would be learned. This seems like a bottleneck to scalability that is not well-addressed.*\n\nEach knowledge key is a trainable variable that can be end-to-end learned with other components of KIAN. Since each knowledge key is a point lying in an embedding space, adding more knowledge policies simply equals adding more points to the embedding space. As a result, KIAN can be easily scaled with a large group of external knowledge policies."}, {"review_id": "AuJ4z04jn0", "reviewer": "Reviewer_2LpZ", "summary": "Humans can learn by aggregating external knowledge from others’ policies of attempting a task. While prior studies in RL have incorporated external knowledge policies for sample efficiency,  there still remains the generalization problem to be solved that agents have difficulties to perform arbitrary combinations and replacements of external policies.\nThe authors propose a new actor architecture for Knowledge-Grounded RL (KGRL),  Knowledge-Inclusive Attention Network (KIAN), which allows free knowledge rearrangement due to embedding-based attentive action prediction.  KIAN addresses entropy imbalance as well. The authors demonstrate in experiments that KIAN outperforms other methods incorporating external knowledge policies under different environmental setups.", "questions": "More experiments directly connected with this challenging problem would more clearly support your claims.", "limitations": "Some insights and analysis about limitations based on the experimental environments more directly connected with the target problem are expected to be included.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors clearly define the problem as how RL can be grounded on any given set of external knowledge policies to achieve knowledge-acquirable, sample-efficient, generalizable, compositional, and incremental properties.\nThe proposed method of KIAN is clearly described. They use knowledge keys and the query performs an attention operation to determine how an agent integrates all policies. \nThe solution of the entropy imbalance issues when integrating external policies are proposed as well.", "weaknesses": "Generally this work has many related works but is tackling the unique challenge problem of fusing knowledge policies with different state and action spaces.\nLimitations of the proposed method is not clear based on the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. *Generally this work has more related works than in the related work section, such as multi-task imitation learning etc,.*\n\nWe understand that other works not listed in the related work section may seem to be addressing a similar problem as KGRL. However, we would like to point out that their central point is quite different from that of KGRL: \n\na) What is our goal? \n\nInvestigate (1) how existing policies can be fused to help an agent learn new tasks more efficiently and (2) how this learning process can be flexible, i.e., by freely reusing and updating the external knowledge-policy set without relearning how to navigate the entire policy set. \n\nb) What is not our goal? \n\nb.1) Hierarchical learning: We specifically discussed the differences between HRL and KGRL in Section 2. While HRL aims to decompose a complex task into a hierarchy of sub-tasks and learn a sub-policy for each sub-task, KGRL aims to address a task by observing and fusing a given set of external policies. \n\nb.2) Multi-task and continual learning: While multi-task RL aims to learn a set of closely related tasks concurrently [49], and continual RL aims to learn a sequence of tasks without forgetting previously learned skills [50,51], KGRL aims to learn a single task more efficiently and flexibly with the help of existing policies from any source while maintaining this efficiency and flexibility to a sequence of tasks. \n\nAccording to our goal, the works listed in Section 2 are better aligned with the scope of this work. However, we will add this discussion to the Appendix in the final paper! \n\n2. *In addition to focusing on the fusion problem of given set of external policies, more explored approaches would be better.\nThe formulation of the problem from broader view of use of external policies would be better.*\n\nWithout a more concrete description of “explored approaches” and “broader view”, we find it difficult to respond to these comments. You seem to suggest a different scope for this work. However, we respectively disagree that a different scope would be better. We would like to restate our motivation for investigating KGRL: Unlike humans, current RL agents lack the ability to efficiently leverage external policies from different sources to help them learn new tasks. In addition, they lack the flexibility to learn with a changeable knowledge-policy set. Without this ability and flexibility, lots of resources will be wasted on (1) relearning the skills that are already developed by someone else and (2) re-exploring how different external policies may help an agent learn a given task. Therefore, we dedicate ourselves to studying KGRL, i.e., how an agent can be knowledge-acquirable, sample efficient, generalizable, compositional, and incremental.\n\n3. *Some insights and analysis about limitations are expected to be included.*\n\nWe have added insights/analyses and limitations of our proposed method! Please refer to “General Response - Ablation Studies and More Analyses” for insights/analyses and “General Response - Limitations” for limitations.\n\n**Reference:**\n\n[49] Vithayathil Varghese, Nelson, and Qusay H. Mahmoud. \"A survey of multi-task deep reinforcement learning.\" Electronics 9.9 (2020): 1363.\n\n[50] Rolnick, David, et al. \"Experience replay for continual learning.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[51] Khetarpal, Khimya, et al. \"Towards continual reinforcement learning: A review and perspectives.\" Journal of Artificial Intelligence Research 75 (2022): 1401-1476.\n"}, {"review_id": "k16BmNgZK3", "reviewer": "Reviewer_CuzY", "summary": "The authors of this work introduce Knowledge-Grounded RL (KGRL), an RL framework that combines multiple knowledge policies to achieve human-like efficiency and flexibility in learning. They propose a new actor architecture called Knowledge-Inclusive Attention Network (KIAN) that enables arbitrary combinations and replacements of external policies through embedding-based attentive action prediction. KIAN also addresses entropy imbalance, a challenge in KGRL, addressing exploration in the environment.", "questions": "See the above mentioned ablation studies.", "limitations": "The limitations have not been discussed by the authors.\nThe above mentioned ablation studies will probably help to identify the shortcomings of the proposed method.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Firstly, the paper addresses a relevant and interesting problem in the field of reinforcement learning (RL) by improving sample efficiency from arbitrary external policies and enabling knowledge transfer skills. \nFurthermore, the paper is a well-written work that effectively conveys the ideas and findings in a clear and concise manner. The authors demonstrate excellent writing skills, employing appropriate terminology and organizing the content in a manner that enhances readability and comprehensibility.\nIn addition, the paper effectively establishes the motivation and position of the study in the existing literature. The authors articulate the significance and relevance of the research problem, demonstrating a strong understanding of the field. They situate their work within the broader scholarly context, highlighting how their study fills a gap in knowledge or builds upon prior research.\nThe methodology employed in the research is clearly described, allowing readers to understand and replicate the study. The authors provide a detailed and comprehensive explanation of the experimental and theoretical approach used, supported by well-designed figures and diagrams. These visual aids enhance the understanding of the methods employed and facilitate better comprehension of the research. Lastly, the authors provide solid mathematical understanding for their proposed method. \n\nCollectively, these strengths highlight the quality of the paper. ts focus on addressing a relevant and interesting problem, combined with its well-written content, clear methodology, positioning in the literature, and mathematical rigor, make it an intersting contribution.", "weaknesses": "The paper has one notable weakness that should be addressed to enhance the overall quality. The evaluation presented in the paper is quite limited. The results obtained for the OpenAI robotics task only offer a slight support for the proposed method. The authors should consider expanding the evaluation to provide a more comprehensive assessment of the proposed method's performance.\nAdditionally, the use of only 3 or 5 seeds in the experiments may be insufficient, especially when considering the large error bars observed in some models. The authors should consider increasing the number of seeds to improve the statistical robustness of the results.\nMoreover, the paper lacks clarity regarding the error bars shown in the plots. It is not explicitly stated what these error bars represent, which hinders the interpretation and understanding of the presented data.\nLastly, the paper would greatly benefit from conducting ablation studies to investigate the effects of various factors on the proposed method's performance. Specifically, the authors could consider performing ablation studies on \n- the influence and distribution of attention/weights of actors, \n- the impact of random/irrelevant policies in $\\mathcal{G}$,\n- the impact of the (near) optimal policy in $\\mathcal{G}$, \n- the impact of a larger set of knowledge policies $\\mathcal{G}$, \n- the effects of different types of knowledge policies, \n- and an investigation of the mentioned entropy balance issue, that the authors specifically address in their method.\nThese ablation studies would provide valuable insights into the individual contributions and impacts of these factors on the overall approach.\n\nOverall, addressing these weaknesses would significantly improve the research paper, clarifying important aspects such as error bars, conducting relevant ablation studies, ensuring consistency in reporting variances, and strengthening the empirical evidence for the proposed method.\n\n### Minor\n- I assume ori-KIAN is KIAN with the original $\\mathcal{G}$, this should be mentioned in the text. \n- l.314 mentions less variance, but tables do not show variances for individual experiments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. *The use of only 3 or 5 seeds in the experiments may be insufficient, especially when considering the large error bars observed in some models. The authors should consider increasing the number of seeds to improve the statistical robustness of the results.*\n\nThank you for the suggestion! We have increased the number of random seeds run for all experiments. Please refer to “General Response - Experimental Significance”.\n\n2. *The paper would greatly benefit from conducting ablation studies to investigate the effects of various factors on the proposed method's performance.*\n\nThank you for the suggestion! We have conducted a series of ablation studies. Please refer to “General Response - Ablation Studies and More Analyses”.\n\n3. *The paper lacks clarity regarding the error bars shown in the plots. It is not explicitly stated what these error bars represent, which hinders the interpretation and understanding of the presented data.*\n\nThank you for pointing out the clarity issue! Each error band in all figures is a 95% confidence interval. In addition, we found that the colors for all algorithms in Figure 3 make it difficult to tell the error bands of KIAN, especially for OpenAI robotic tasks. Thus, we changed the colors of all curves in Figure 5. We will update the explanation of the error bands and figures in the final paper.\n\n4. *The results obtained for the OpenAI robotics task only offer a slight support for the proposed method.*\n\nActually, for Pick-and-Place and Push in Figure 3 (Figure 5) as well as Push->Pick-and-Place in Figure 4, KIAN converges much faster and has much smaller error bars compared to all other baselines. Also, in Tables 1 and 2 (zero-shot transfer experiments), KIAN’s performance is generally the best or very close to the best performance for OpenAI robotic tasks. Finally, in Appendix E, we showed that KIAN performs better in zero-shot transfer experiments with small variances for all OpenAI robotic tasks. We believe that these results suggest that KIAN is also effective for continuous-control tasks.\n\n5. *I assume ori-KIAN is KIAN with the original , this should be mentioned in the text.*\n\nThank you for pointing out this clarity issue! Yes, ori-KIAN is KIAN without using previously learned query and keys, which is the same setting as used in Figure 3 (Figure 5). We will clarify this in the final paper.\n\n6. *l.314 mentions less variance, but tables do not show variances for individual experiments*\n\nThank you for pointing out this clarity issue! We were limited by space when submitting the original paper, so we moved the tables with variances to Appendix E. We will clarify this in the final paper.\n\n7. *The limitations have not been discussed by the authors. The above mentioned ablation studies will probably help to identify the shortcomings of the proposed method.*\n\nWe have added limitations of our proposed method! Please refer to “General Response - Limitations”. \n\n8. *The influence and distribution of attention/weights of actors:* \n\nFigure 6 in the additional-result document shows the weights of each knowledge policy after training. The results demonstrate different behaviors of KIAN in discrete decision-making and continuous control tasks.\n\n8.a) Discrete decision-making: An agent learns to attend to different knowledge policies according to the current situation. Specifically, in a DoorKey-8x8 environment, the agent chooses to sequentially “pick up the key”, “open the door”, “search for the goal”, and “reach the goal”. Note that the attention to the inner policy does not disappear throughout the entire episode. This shows that the inner policy helps an agent explore the environment.\n\n8.b) Continuous control: An agent learns to attend to only the inner knowledge policy. This result shows that KIAN enables effective knowledge transfer between external and inner policies. We believe this difference from the behaviors in discrete decision-making can be attributed to (1) how an action is sampled in continuous control tasks and (2) the suboptimality of external policies: Since an action is sampled by Gumbel softmax, only the action suggested by one knowledge policy will be applied, even though in probability, all knowledge policies are fused together. As a result, the agent may find it more efficient to let the inner policy first copy the useful skills from suboptimal external policies and focus on the inner policy to complete a task."}, {"review_id": "hlBoXzzccc", "reviewer": "Reviewer_K32E", "summary": "This paper introduces KIAN, a method for leveraging multiple sub-optimal policies to solve reinforcement learning tasks. The paper starts out by introducing a new Knowledge-Grounded MDP, which adds a set of external knowledge policies to the traditional MDP framework. \n\nTo leverage these external knowledge policies, KIAN learns an embedding for each knowledge policy (including an inner knowledge policy for the current agent). Then at each step, the current state is mapped to a query embedding that can be used to find the policy that is best equipped to take an action at that timestep.", "questions": "*  I don't fully understand the incremental property. In the introduction. the authors state that \"Humans do not need to relearn how to navigate the entire knowledge set from scratch when they remove outdated strategies or add new ones.\" But then Definition 3.5 states that an incremental agent has two properties:\n    1. The agent can use different knowledge policies to solve a single KGMDP\n    2. Given a sequence of KGMDPs, the agent can solve them with different knowledge sets.\n\n    * In its current form, definition 3.5 does not specify the case where an agent adds or removes a specific policy from its knowledge set. I think a condition should be added to highlight how an incremental agent should be able to leverage a sequence of knowledge sets that differ by one or more policies.\n\n\n* How do the initial external knowledge policies perform on the given tasks?", "limitations": "The authors discussed broader impact and future work in Appendix D, but no limitations were addressed. Could the authors please add limitations to either Appendix D or the conclusion?", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper is structured well and is easy to read.\n* The authors demonstrated great attention to detail by motivating the theorems and equations with intuition before defining them concretely.\n*  The paper is very technically sound.", "weaknesses": "* > In such a case, whenever the knowledge set is updated by adding or replacing policies, prior methods require relearning the entire multi-policy fusion process, even if the current task is similar to the previous one. This is because their designs of knowledge representations are intertwined with the knowledge-fusing mechanism, which restricts the number of policies in the knowledge set from being changed.\n\n    * It would be good to point to specific prior works that suffer from this problem so the reader can build intuition.\n\n* There are no visualizations of the tasks used for evaluation. Adding pictures of the environments would help readers understand what the agent needs to do.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. *\"In such a case, whenever the knowledge set is updated by adding or replacing policies, prior methods require relearning the entire multi-policy fusion process, even if the current task is similar to the previous one. This is because their designs of knowledge representations are intertwined with the knowledge-fusing mechanism, which restricts the number of policies in the knowledge set from being changed.\"*\n*It would be good to point to specific prior works that suffer from this problem so the reader can build intuition.*\n\nThank you for the suggestion! We will add citations (e.g., for KoGuN and A2T) after the statement.\n\n2. *There are no visualizations of the tasks used for evaluation. Adding pictures of the environments would help readers understand what the agent needs to do.*\n\nThank you for the suggestion! We provided some screenshots of the environments used for evaluation in Figures 6 and 10 in the additional-result document and will add them to the final paper.\n\n3. *I don't fully understand the incremental property. In the introduction. the authors state that \"Humans do not need to relearn how to navigate the entire knowledge set from scratch when they remove outdated strategies or add new ones.\" But then Definition 3.5 states that an incremental agent has two properties:*\n\n3.a) The agent can use different knowledge policies to solve a single KGMDP\n\n3.b) Given a sequence of KGMDPs, the agent can solve them with different knowledge sets.\n\n*In its current form, definition 3.5 does not specify the case where an agent adds or removes a specific policy from its knowledge set. I think a condition should be added to highlight how an incremental agent should be able to leverage a sequence of knowledge sets that differ by one or more policies.*\n\nThank you for the suggestion! We assumed changing $\\mathcal{G}$ includes the operations of adding/removing knowledge policies. We will add concrete conditions of these operations in Definition 3.5.\n\n4. *How do the initial external knowledge policies perform on the given tasks?*\n\nThe initial external knowledge policies barely succeeded in most of the tasks we evaluated. These results are plotted in Figure 3 with the label “BC”.\n\n5. *The authors discussed broader impact and future work in Appendix D, but no limitations were addressed. Could the authors please add limitations to either Appendix D or the conclusion?*\n\nYes! We have discussed the limitations of this work. Please refer to “General Response - Limitations”. "}]

Author rebuttal:
We thank the reviewers for their insightful feedback! We address the shared reviewers’ comments below and will incorporate all feedback in the final paper.

### Experimental Significance
(Reviewer 7i4w and CuzY)

We have increased the number of random seeds to 10 for each experiment within the given time frame. Yet, we are still running with more random seeds. Also, we changed the colors of all curves in Figure 5 in the additional-result document for easier to read the error bands. Each error band in all figures is a 95% confidence interval. We will update the figures and explanation of the error bands in the final paper.

The results in Figure 5 show that even after adding more random seeds, KIAN converges much faster and reaches higher rewards for all tasks. In addition, KIAN’s error bands are small, indicating that its improved performance is statistically significant. 

We observe similar results when running more random seeds for Figure 4. We apologize that due to time and resource limitations, we are not able to provide an updated plot for Figure 4 on time. Yet, we will update the results in the final paper. 

### Ablation Studies and More Analyses
(Reviewer 2LpZ and CuzY)

* The impact of random/irrelevant policies in $\mathcal{G}$: Figure 7 in the additional-result document shows the learning curves of KIAN trained with 3 relevant, 6 (3 relevant + 3 irrelevant), and 9 (3 relevant + 6 irrelevant) external policies for MiniGrid Unlock. The results demonstrate that adding redundant/irrelevant knowledge policies slightly slows down the convergence speed but still reaches high rewards with small error bands. This slight convergence-speed decrease is quite as expected since the agent needs to (1) learn to ignore the policies that are not helpful for the task and (2) navigate through a larger set of external knowledge policies. 

* The impact of the (near) optimal policy in $\mathcal{G}$: Figure 8 in the additional-result document shows the results of running FetchReach-v1 with an optimal external knowledge policy. We did this ablation study on FetchReach-v1 since we can only obtain the optimal policy for this environment without doing further training. 

    The results demonstrate that with optimal external policies, SAC-RL, SAC-KoGuN, and SAC-KIAN achieve similar performance, but SAC-KIAN still performs slightly better than others. 

* The impact of a larger set of knowledge policies $\mathcal{G}$: In Figure 7 in the additional-result document, we can also see the effects of learning with a larger set of external policies. The results show that the convergence speed slows down when the knowledge set is expanded with more policies that are irrelevant to the task. 

    We believe that one of the reasons for slower convergence speed is that the agent needs to navigate through more policies before it knows which ones are useful for the task. Therefore, when the knowledge-policy set is really large, KGRL methods, including KoGuN, A2T, and KIAN, are not guaranteed to be more efficient than RL methods. We will also discuss this point in the limitations. 

* The effects of different types of knowledge policies: Actually, KIAN’s results in Figure 4 (in the original paper) are run with an external knowledge set that includes two different types of knowledge policies: if-else-based programs and neural networks. The results demonstrate that mixing different types of knowledge policies will not hurt performance at all. This observation is as we expected since KIAN only cares about the state-action mappings provided by each knowledge policy instead of how this mapping is generated. 

* An investigation of the mentioned entropy balance issue: Figure 9 in the additional-result document shows the results of KIAN with and without the entropy imbalance issue addressed. The results show that without addressing the issue with equation (9), KIAN cannot fully benefit from the guidance provided by external policies. In addition, KIAN may perform even worse than RL with no external knowledge policies (see the results of FetchPickAndPlace and FetchPush). The reason is provided by Proposition 4.3. 

    These results demonstrate that (1) entropy imbalance is indeed an issue for KGRL and (2) our proposed method can largely mitigate this issue. 



### Limitations
(Reviewer 2LpZ, CuzY, K32E) 

Here are the limitations we will summarize in Conclusion/Appendix D: 
* If the external knowledge set is very large and contains irrelevant policies, the efficiency of KIAN, and other existing KGRL methods, may decrease. This is discussed in “General Response - 2.4”. 
* The current design of KIAN requires external and inner policies to share the same state and action space. Therefore, if a policy for Task A includes a skill that can benefit Task B, but Task A and B have different state and action space, the policy cannot be used as an external policy for Task B. This can limit the number of existing policies that can be used to help learn a new task.


Meta-review: This paper proposes a Knowledge-Grounded RL for integrating knowledge (in the form of policies) into a policy to learn new tasks efficiently. The paper outlines five desirable human-like properties they desire in their agents: knowledge-acquirable, sample-efficient, generalizable, compositional, and incremental. The authors have shown the value of this work via additional experiments, and the reviewers have not found technical flaws in the work. Most reviewers appreciate how the work studies the entropy imbalance problem in entropy-regularized KGRL.
Decision: Accept (poster)