Title: Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Deep Reinforcement Learning

Sample Efficiency

Generalizability

Multi-Policy Decision Making

Multi-Policy Continuous Control
Abstract: Reinforcement learning (RL) agents have long sought to approach the efficiency of human learning. Humans are great observers who can learn by aggregating external knowledge from various sources, including observations from others' policies of attempting a task. Prior studies in RL have incorporated external knowledge policies to help agents improve sample efficiency. However, it remains non-trivial to perform arbitrary combinations and replacements of those policies, an essential feature for generalization and transferability. In this work, we present Knowledge-Grounded RL (KGRL), an RL paradigm fusing multiple knowledge policies and aiming for human-like efficiency and flexibility. We propose a new actor architecture for KGRL, Knowledge-Inclusive Attention Network (KIAN), which allows free knowledge rearrangement due to embedding-based attentive action prediction. KIAN also addresses entropy imbalance, a problem arising in maximum entropy KGRL that hinders an agent from efficiently exploring the environment, through a new design of policy distributions. The experimental results demonstrate that KIAN outperforms alternative methods incorporating external knowledge policies and achieves efficient and flexible learning. Our implementation is available at https://github.com/Pascalson/KGRL.git .

Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning

Abstract

Reinforcement learning (RL) agents have long sought to approach the efficiency of human learning. Humans are great observers who can learn by aggregating external knowledge from various sources, including observations from others' policies of attempting a task. Prior studies in RL have incorporated external knowledge policies to help agents improve sample efficiency. However, it remains non-trivial to perform arbitrary combinations and replacements of those policies, an essential feature for generalization and transferability. In this work, we present Knowledge-Grounded RL (KGRL), an RL paradigm fusing multiple knowledge policies and aiming for human-like efficiency and flexibility. We propose a new actor architecture for KGRL, Knowledge-Inclusive Attention Network (KIAN), which allows free knowledge rearrangement due to embedding-based attentive action prediction. KIAN also addresses entropy imbalance, a problem arising in maximum entropy KGRL that hinders an agent from efficiently exploring the environment, through a new design of policy distributions. The experimental results demonstrate that KIAN outperforms alternative methods incorporating external knowledge policies and achieves efficient and flexible learning. Our implementation is available at https://github.com/Pascalson/KGRL.git .

1 Introduction

Reinforcement learning (RL) has been effectively used in a variety of fields, including physics [7, 35] and robotics [15, 30]. This success can be attributed to RL's iterative process of interacting with the environment and learning a policy to get positive feedback. Despite being influenced by the learning process of infants [32], the RL process can require a large number of samples to solve a task [1], indicating that the learning efficiency of RL agents is still far behind that of humans.

What learning capabilities do humans possess, yet RL agents still missing? Studies in social learning [4] have demonstrated that humans often observe the behavior of others in diverse situations and utilize those strategies as external knowledge to accelerate their own exploration of solutionspace. This type of learning is very flexible for humans since they can freely reuse and update the knowledge they already possess. The followings are the five properties (the last four have been mentioned in [14]) that summarize the efficiency and flexibility of human learning. [KnowledgeAcquirable] : Humans can develop their strategies by observing others. [Sample-Efficient] : Humans require fewer interactions with the environment to solve a task by learning from external knowledge. [Generalizable] : Humans can apply previously observed strategies, whether developed internally or provided externally, to unseen tasks. [Compositional] : Humans can combine strategies from multiple sources to form their knowledge set. [Incremental] : Humans do not need to relearn how to navigate the entire knowledge set from scratch when they remove outdated strategies or add new ones.

Policy

Observed

from

Solved

for

Joy

Policy

Observed

from

Solved

for

Joy

Myself

if ...

else ...

o°

记

O：

B

Possessing all five learning properties remains challenging for RL agents. Previous work has endowed an RL agent with the ability to learn from external knowledge (knowledge-acquirable) and mitigate sample inefficiency [21, 25, 27, 36], where the knowledge focused in this paper is state-action mappings (full definition in Section 3), including pre-collected demonstrations or policies. Among those methods, some have also allowed agents to combine policies in different forms to predict optimal actions (compositional) [25, 27]. However, these approaches may not be suitable for incremental learning, in which an agent learns a sequence of tasks using one expandable knowledge set. In such a case, whenever the knowledge set is updated by adding or replacing policies, prior methods, e.g., [27, 36], require relearning the entire multi-policy fusion process, even if the current task is similar to the previous one. This is because their designs of knowledge representations are intertwined with the knowledge-fusing mechanism, which restricts changing the number of policies in the knowledge set.

To this end, our goal is to enhance RL grounded on external knowledge policies with more flexibility. We first introduce Knowledge-Grounded Reinforcement Learning (KGRL) , an RL paradigm that seeks to find an optimal policy of a Markov Decision Process (MDP) given a set of external policies as illustrated in Figure 1. We then formally define the knowledge-acquirable, sample-efficient, generalizable, compositional, and incremental properties that a well-trained KGRL agent can possess.

We propose a simple yet effective actor model, Knowledge-Inclusive Attention Network (KIAN) , for KGRL. KIAN consists of three components: (1) an internal policy that learns a self-developed strategy, (2) embeddings that represent each policy, and (3) a query that performs embedding-based attentive action prediction to fuse the internal and external policies. The policy-embedding and query design in KIAN is crucial, as it enables the model to be incremental by unifying policy representations and separating them from the policy-fusing process. Consequently, updating or adding policies to KIAN has minimal effect on its architecture and does not require retraining the entire network. Additionally, KIAN addresses the problem of entropy imbalance in KGRL, where agents tend to choose only a few sub-optimal policies from the knowledge set. We provide mathematical evidence that entropy imbalance can prevent agents from exploring the environment with multiple policies. Then we introduce a new approach for modeling external-policy distributions to mitigate this issue.

Through experiments on grid navigation [5] and robotic manipulation [24] tasks, KIAN outperforms alternative methods incorporating external policies in terms of sample efficiency as well as the ability to do compositional and incremental learning. Furthermore, our analyses suggest that KIAN has better generalizability when applied to environments that are either simpler or more complex.

Our contributions are:

We introduce KGRL, an RL paradigm studying how agents learn with external policies while being knowledge-acquirable, sample-efficient, generalizable, compositional, and incremental.

We propose KIAN, an actor model for KGRL that fuses multiple knowledge policies with better flexibility and addresses entropy imbalance for more efficient exploration.

We demonstrate in experiments that KIAN outperforms other methods incorporating external knowledge policies under different environmental setups.

2 Related Work

A popular line of research in RL is to improve sample efficiency with demonstrations (RL from demonstrations; RLfD). Demonstrations are examples of completing a task and are represented as state-action pairs. Previous work has leveraged demonstrations by introducing them into the policy-update steps of RL [8, 11, 21, 23, 28, 34]. For example, Nair et al. [21] adds a buffer of demonstrations to the RL framework and uses the data sampled from it to calculate a behaviorcloning loss. This loss is combined with the regular RL loss to make the policy simultaneously imitate demonstrations and maximize the expected return. RLfD methods necessitate an adequate supply of high-quality demonstrations to achieve sample-efficient learning, which can be time-consuming. In addition, they are low-level representations of a policy. Consequently, if an agent fails to extract a high-level strategy from these demonstrations, it will merely mimic the actions without acquiring a generalizable policy. In contrast, our proposed KIAN enables an agent to learn with external policies of arbitrary quality and fuse them by evaluating the importance of each policy to the task . Thus, the agent must understand the high-level strategies of each policy rather than only imitating its actions.

Another research direction in RL focuses on utilizing sub-optimal external policies instead of demonstrations to improve sample efficiency [25, 27, 36]. For instance, Zhang et al. [36] proposed Knowledge-Guided Policy Network (KoGuN) that learns a neural network policy from fuzzy-rule controllers. The neural network concatenates a state and all actions suggested by fuzzy-rule controllers as an input and outputs a refined action. While effective, this method puts restrictions on the representation of a policy to be a fuzzy logic network. On the other hand, Rajendran et al. [27] presented A2T (Attend, Adapt, and Transfer), an attentive deep architecture that fuses multiple policies and does not restrict the form of a policy. These policies can be non-primitive, and a learnable internal policy is included. In A2T, an attention network takes a state as an input and outputs the weights of all policies. The agent then samples an action from the fused distribution based on these weights. The methods KoGuN and A2T are most related to our work. Based on their success, KIAN further relaxes their requirement of retraining for incremental learning since both of them depend on the preset number of policies. Additionally, our approach mitigates the entropy imbalance issue, which can lead to inefficient exploration and was not addressed by KoGuN and A2T.

There exist other RL frameworks, such as hierarchical RL (HRL), that tackle tasks involving multiple policies. However, these frameworks are less closely related to our work compared to the previously mentioned methods. HRL approaches aim to decompose a complex task into a hierarchy of sub-tasks and learn a sub-policy for each sub-task [2, 6, 13, 16-18, 20, 25, 31, 33]. On the other hand, KGRL methods, including KoGuN, A2T, and KIAN, aim to address a task by observing a given set of external policies. These policies may offer partial solutions, be overly intricate, or even have limited relevance to the task at hand. Furthermore, HRL methods typically apply only one sub-policy to the environment at each time step based on the high-level policy, which determines the sub-task the agent is currently addressing. In contrast, KGRL seeks to simultaneously apply multiple policies within a single time step by fusing them together.

3 Problem Formulation

Our goal is to investigate how RL can be grounded on any given set of external knowledge policies to achieve knowledge-acquirable, sample-efficient, generalizable, compositional, and incremental properties. We refer to this RL paradigm as Knowledge-Grounded Reinforcement Learning (KGRL).

A KGRL problem is a sequential decision-making problem that involves an environment, an agent, and a set of external policies. It can be mathematically formulated as a Knowledge-Grounded Markov Decision Process (KGMDP), which is defined by a tuple M k = ( S , A , T , R, ρ, γ, G ) , where S is the state space, A is the action space, T : S × A × S → R is the transition probability distribution, R is the reward function, ρ is the initial state distribution, γ is the discount factor, and G is the set of external knowledge policies. An external knowledge set G contains n knowledge policies, G = { π g 1 , . . . , π g n } . Each knowledge policy is a function that maps from the state space to the action space, π g j ( ·|· ) : S → A , ∀ j = 1 , . . . , n . A knowledge mapping is not necessarily designed for the original Markov Decision Process (MDP), which is defined by the tuple M = ( S , A , T , R , ρ, γ ) . Therefore, applying π g j to M may result in a poor expected return.

The goal of KGRL is to find an optimal policy π ∗ ( ·|· ; G ) : S → A that maximizes the expected return: E s 0 ∼ ρ, T ,π ∗ [ ∑ T t =0 γ t R t ] . Note that M k and M share the same optimal value function, V ∗ ( s ) = max π ∈ Π E T ,π [ ∑ ∞ k =0 γ k R t + k +1 | s t = s ] , if they are provided with the same policy class Π .

A well-trained KGRL agent can possess the following properties: knowledge-acquirable, sampleefficient, generalizable, compositional, and incremental. Here we formally define these properties.

Definition 3.1 (Knowledge-Acquirable) . An agent can acquire knowledge internally instead of only following G . We refer to this internal knowledge as an inner policy and denote it as π in ( ·|· ) : S → A .

Definition 3.2 (Sample-Efficient) . An agent requires fewer samples to solve for M k than for M .

Definition 3.3 (Generalizable) . A learned policy π ( ·|· ; G ) can solve similar but different tasks.

Definition 3.4 (Compositional) . Assume that other agents have solved for m KGMDPs, M 1 k , . . . , M m k , with external knowledge sets, G 1 , . . . , G m , and inner policies, π 1 in , . . . , π m in . An agent is compositional if it can learn to solve a KGMDP M ∗ k with the external knowledge set G ∗ ⊆ ⋃ m i =1 G i ∪ { π 1 in , . . . , π m in } .

Definition 3.5 (Incremental) . An agent is incremental if it has the following two abilities: (1) Given a KGMDP M k for the agent to solve within T timesteps. The agent can learn to solve M k with the external knowledge sets, G 1 , . . . , G T , where G t , t ∈ { 1 , . . . , T } , is the knowledge set at time step t , and G t can be different from one another. (2) Given a sequence of KGMDPs M 1 k , . . . , M m k , the agent can solve them with external knowledge sets, G 1 , . . . , G m , where G i , i ∈ { 1 , . . . , m } , is the knowledge set for task i , and G i can be different from one another.

4 Knowledge-Inclusive Attention Network

Wepropose Knowledge-Inclusive Attention Network (KIAN) as an actor for KGRL. KIAN can be end-to-end trained with various RL algorithms. Illustrated in Figure 2, KIAN comprises three components: an inner actor, knowledge keys, and a query. In this section, we first describe the architecture of KIAN and its actionprediction operation. Then we introduce entropy imbalance, a problem that emerges in maximum entropy KGRL, and propose modified policy distributions for KIAN to alleviate this issue.

4.1 Model Architecture

Inner Actor. The inner actor serves the same purpose as an actor in regular RL, representing the inner knowledge learned by the agent through interactions with the environment. In

KIAN

Inner Actor

Knowledge

(g 1 )

k

in

u

Knowledge

(g 2 )

Knowledge

(g 3 )

action (

a

t )

state (

s

t )

Environment

KIAN, the inner actor, denoted as π in ( ·|· ; θ ) : S → A , is a learnable function approximator with parameter θ . The presence of the inner actor in KIAN is crucial for the agent to be capable of acquiring knowledge, as it allows the agent to develop its own strategies. Therefore, even if the external knowledge policies in G are unable to solve a particular task, the agent can still discover an optimal solution.

Knowledge Keys. In KIAN, we introduce a learnable embedding vector for each knowledge policy, including π in and π g 1 , . . . , π g n , in order to create a unified representation space for all knowledge policies. Specifically, for each knowledge mapping π in or π g j ∈ G , we assign a learnable d k -dimensional vector as its key (embedding): k in ∈ R d k or k g j ∈ R d k ∀ j ∈ { 1 , . . . , n } . It is important to note that these knowledge keys, k e , represents the entire knowledge mapping π e , ∀ e ∈ { in, g 1 , . . . , g n } . Thus, k e is independent of specific states or actions. These knowledge keys and the query will perform an attention operation to determine how an agent integrates all policies.

Our knowledge-key design is essential for an agent to be compositional and incremental. By unifying the representation of policies through knowledge keys, we remove restrictions on the form of a

knowledge mapping. It can be any form, such as a lookup table of state-action pairs (demonstrations) [21], if-else-based programs, fuzzy logics [36], or neural networks [25, 27]. In addition, the knowledge keys are not ordered, so π g 1 , . . . , π g n in G and their corresponding k g 1 , . . . , k g n can be freely rearranged. Finally, since a knowledge policy is encoded as a key independent of other knowledge keys in a joint embedding space, replacing a policy in G means replacing a knowledge key in the embedding space. This replacement requires no changes in the other part of KIAN's architecture. Therefore, an agent can update G anytime without relearning a significant part of KIAN.

Query. The last component in KIAN, the query , is a function approximator that generates d k -dimensional vectors for knowledge-policy fusion. The query is learnable with parameter ϕ and is state-dependent, so we denote it as Φ( · ; ϕ ) : S → R d k . Given a state s t ∈ S , the query outputs a d k -dimensional vector u t = Φ( s t ; ϕ ) ∈ R d k , which will be used to perform an attention operation with all knowledge keys. This operation determines the weights of policies when fusing them.

4.2 Embedding-Based Attentive Action Prediction

The way to predict an action with KIAN and a set of external knowledge policies, G , is by three steps: (1) calculating a weight for each knowledge policy using an embedding-based attention operation, (2) fusing knowledge policies with these weights, and (3) sampling an action from the fused policy.

Embedding-Based Attention Operation. Given a state s t ∈ S , KIAN predicts a weight for each knowledge policy as how likely this policy will suggest a good action . These weights can be computed by the dot product between the query and knowledge keys as:

where c t,in ∈ R and c t,g j ∈ R are normalization factors, for example, if c t,g j = ∥ Φ( s t ; ϕ ) ∥ 2 ∥ k g j ∥ 2 , then w t,g j turns out to be the cosine similarity between Φ( s t ; ϕ ) and k g j . We refer to this operation as an embedding-based attention operation since the query evaluates each knowledge key (embedding) by equation (1) to determine how much attention an agent should pay to the corresponding knowledge policy. If w t,in is larger than w t,g j , the agent relies more on its self-learned knowledge policy π in ; otherwise, the agent depends more on the action suggested by the knowledge policy π g j . Note that the computation of one weight is independent of other knowledge keys, so changing the number of knowledge policies will not affect the relation among all remaining knowledge keys.

Action Prediction for A Discrete Action Space. An MDP (or KGMDP) with a discrete action space usually involves choosing from d a ∈ N different actions, so each knowledge policy maps from a state to a d a -dimensional probability simplex , π in : S → ∆ d a , π g j : S → ∆ d a ∀ j = 1 , . . . , n . When choosing an action given a state s t ∈ S , KIAN first predicts π ( ·| s t ) ∈ ∆ d a ⊆ R d a with the weights, ˆ w in , ˆ w g 1 , . . . , ˆ w g n :

The final action is sampled as a t ∼ π ( ·| s t ) , where the i -th element of π ( ·| s t ) represents the probability of sampling the i -th action.

Action Prediction for A Continuous Action Space. Each knowledge policy for a continuous action space is a probability distribution that suggests a d a -dimensional action for an agent to apply to the task. As prior work [25], we model each knowledge policy as a multivariate normal distribution, π in ( ·| s t ) = N ( µ t,in , σ 2 t,in ) , π g j ( ·| s t ) = N ( µ t,g j , σ 2 t,g j ) ∀ j ∈ { 1 , . . . , n } , where µ t,in ∈ R d a and µ t,g j ∈ R d a are the means, and σ 2 t,in ∈ R d a ≥ 0 and σ 2 t,g j ∈ R d a ≥ 0 are the diagonals of the covariance matrices. Note that we assume each random variable in an action is independent of one another.

A continuous policy fused as equation (3) becomes a mixture of normal distributions. To sample an action from this mixture of distributions without losing the important information provided by each distribution, we choose only one knowledge policy according to the weights and sample an action from it. We first sample an element from the set

{ in, g 1 , . . . , g n } according to the weights, { ˆ w t,in , ˆ w t,g 1 , . . . , ˆ w t,g n } , using Gumbel softmax [12]: e ∼ gumbel_softmax ([ ˆ w t,in , ˆ w t,g 1 , . . . , ˆ w t,g n ] ⊤ ) , in order to make KIAN differentiable everywhere. Then given a state s t ∈ S , an action is sampled from the knowledge policy, a t ∼ π e ( ·| s t ) , using the reparameterization trick.

However, fusing multiple policies as equation (3) will make an agent biased toward a small set of knowledge policies when exploring the environment in the context of maximum entropy KGRL.

4.3 Exploration in KGRL

Maximizing entropy is a commonly used approach to encourage exploration in RL [9, 10, 37]. However, in maximum entropy KGRL, when the entropy of policy distributions are different from one another, it leads to the problem of entropy imbalance . Entropy imbalance is a phenomenon in which an agent consistently selects only a single or a small set of knowledge policies. We show this in math by first revisiting the formulation of maximum entropy RL. In maximum entropy RL, an entropy term is added to the standard RL objective as π ∗ = argmax π ∑ t E ( s t , a t ∼ π ) [ R ( s t , a t ) + αH ( π ( ·| s t ))] [9, 10], where α ∈ R is a hyperparameter, and H ( · ) represents the entropy of a distribution. By maximizing αH ( π ( ·| s t )) , the policy becomes more uniform since the entropy of a probability distribution is maximized when it is a uniform distribution [19]. With this in mind, we show that in maximum entropy KGRL, some of the weights in { ˆ w t,in , ˆ w t,g 1 , . . . , ˆ w t,g n } might always be larger than others. We provide the proofs of all propositions in Appendix A.

Proposition 4.1 (Entropy imbalance in discrete decision-making) . Assume that a d a -dimensional probability simplex π ∈ ∆ d a is fused by { π 1 , . . . , π m } and { ˆ w 1 , . . . , ˆ w m } following equation (3), where π j ∈ ∆ d a , ˆ w j ≥ 0 ∀ j ∈ { 1 , . . . , m } and ∑ m j =1 ˆ w j = 1 . If the entropy of π is maximized and ∥ π 1 ∥ ∞ ≪∥ π 2 ∥ ∞ , ∥ π 1 ∥ ∞ ≪∥ π 3 ∥ ∞ , . . . , ∥ π 1 ∥ ∞ ≪∥ π m ∥ ∞ , then ˆ w 1 → 1 .

We show in Proposition A.1 that if π 1 is more uniform than π j , then ∥ π 1 ∥ ∞ < ∥ π j ∥ ∞ .

Proposition 4.2 (Entropy imbalance in continuous control) . Assume a one-dimensional policy distribution π is fused by

If the variance of π is maximized, and σ 2 1 ≫ σ 2 2 and σ 2 1 ≫ ( µ 1 -µ 2 ) 2 , then ˆ w 1 → 1 .

We can also infer from Proposition 4.2 that the variance of π defined in equation (4) depends on the distance between µ 1 and µ 2 , which leads to Proposition 4.3.

Proposition 4.3 (Distribution separation in continuous control) . Assume a one-dimensional policy distribution π is fused by equation (4). If ˆ w 1 , ˆ w 2 , σ 2 1 , and σ 2 2 are fixed, then maximizing the variance of π will increase the distance between µ 1 and µ 2 .

Proposition 4.1, 4.2, and 4.3 indicate that in maximum entropy KGRL, (1) the agent will pay more attention to the policy with large entropy, and (2) in continuous control, an agent with a learnable internal policy will rely on this policy and separate it as far away as possible from other policies. The consistently imbalanced attention prevents the agent from exploring the environment with other policies that might provide helpful suggestions to solve the task. Furthermore, in continuous control, the distribution separation can make π perform even worse than learning without any external knowledge. The reason is that external policies, although possibly being sub-optimal for the task, might be more efficient in approaching the goal, and moving away from those policies means being less efficient when exploring the environment.

4.4 Modified Policy Distributions

Proposition 4.1 and 4.2 show that fusing multiple policies with equation (3) can make a KGRL agent rely on a learnable internal policy for exploration. However, the uniformity of the internal policy is often desired since it encourages exploration in the state-action space that is not covered by external policies. Therefore, we keep the internal policy unchanged and propose methods to modify external policy distributions in KIAN to resolve the entropy imbalance issue. We provide the detailed learning algorithm of KGRL with KIAN in Appendix A.6.

Discrete Policy Distribution. We modify a fusion of discrete policy distributions in equation (3) as

where β t,g j ∈ R is a state-and-knowledge dependent variable that scales π g j ( ·| s t ) to change its uniformity after passing through softmax . If the value of β t,g j decreases, the uniformity, i.e., the entropy, of softmax ( β t,g j π g j ( ·| s t )) increases. By introducing β t,g j , the entropy of knowledge policies becomes adjustable, resulting in reduced bias towards the internal policy during exploration.

Continuous Action Probability. We modify the probability of sampling a t ∈ R d a from a continuous π ( ·| s t ) in equation (3) as

where a t,in ∼ π in ( ·| s t ) and µ t,g j ∈ R d a is the mean of π g j ( ·| s t ) . We show in the next proposition that equation (8) is an approximation of

which is the exact probability of sampling a t ∈ R d a from a continuous π ( ·| s t ) in equation (3).

Proposition 4.4 (Approximation of a mixture of normal distributions) . If the following three inequalities hold for µ t,in , µ t,g 1 , . . . , µ t,g n , and a t,in : ∥ µ t,in -µ t,g j ∥ 2 < min { γ t,in , γ t,g j } , ∥ a t,in -µ t,in ∥ 2 < min { γ t,in , γ t,g j } , and ∥ a t,in -µ t,g j ∥ 2 < γ t,g j , ∀ j ∈ { 1 , . . . , n } , where γ t,in = 1 / (2 π in ( µ t,in | s t )) and γ t,g j = 1 / (2 π g j ( µ t,g j | s t )) , then equation (9) for a real-valued action a t sampled from KIAN can be approximated by

In addition, equation (8) is a lower bound of equation (10).

With equation (8), we can show that maximizing the variance of π ( ·| s t ) will not separate the policy distributions. Hence, an agent can refer to external policies for efficient exploration and learn its own refined strategy based on them.

Proposition 4.5 (Maximized variance's independence of the distance between means) . Assume a one-dimensional policy π is fused by equation (4). If π ( a | s ) is approximated as equation (8), and the three inequalities in Proposition 4.4 are satisfied, then maximizing the variance of π ( ·| s ) will not affect the distance between µ 1 and µ 2 .

5 Experiments

We evaluate KIAN on two sets of environments with discrete and continuous action spaces: MiniGrid [5] and OpenAI-Robotics [24]. Through experiments, we answer the following four questions: [Sample Efficiency] Does KIAN require fewer training samples to solve a task than other externalpolicy-inclusive methods? [Generalizability] Can KIAN trained on one task be directly used to solve another task? [Compositional and Incremental Learning] Can KIAN combine previously learned knowledge keys and inner policies to learn a new task? After adding more external policies to G , can most of the components from a trained KIAN be reused for learning?

For comparison, we implement the following five methods as our baselines: behavior cloning (BC) [3], RL [10, 29], RL+BC [21], KoGuN [36], and A2T [27]. KoGuN and A2T are modified to be compositional and applicable in both discrete and continuous action spaces. Moreover, all methods (BC, RL+BC, KoGuN, A2T, and KIAN) are equipped with the same initial external knowledge set, G init , for each task. This knowledge set comprises sub-optimal if-else-based programs that cannot complete a task themselves, e.g., pickup_a_key or move_forward_to_the_goal . G init will be expanded with learned policies in compositional- and incremental-learning experiments. We provide the experimental details in Appendix B.

BC

PPO

PPO+BC

PPO+KoGuN

PPO+A2T

PPO+KIAN

Empty-Random-5x5

Unlock

DoorKey-5x5

DoorKey-8x8

1.0

1.0

1.0

Reward

Reward

Reward

Reward

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2.5

5.0

7.5

0

1

2

3

0.0

2.55.0

7.5

0

1

2

3

Timesteps

1e4

Timesteps

1e5

Timesteps

1e4

Timesteps

1e6

Dynamic-0bstacles-16x16

LavaCrossingS9N2

MultiRoom-N4-S5

KeyCorridorS3R2

0.6

0.75

Reward

Reward

0

0.5

0.2

-1

0.0

0.0

0.00

0

1

2

3

0

2

4

0

2

4

0.0

0.5

1.0

Timesteps

1e5

Timesteps

1e6

Timesteps

1e5

Timesteps

1e6

BC

SAC

SAC+BC

SAC+KoGuN

SAC+A2T

SAC+KIAN

Pick-and-Place

Push

Slide

1.0

1.0

0.75

Reward

Reward

Reward

0.50

0.5

0.5

0.25

0.0

0.0

0.00

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

Timesteps

1e6

Timesteps

1e6

Timesteps

1e6

5.1 Sample Efficiency and Generalizability

We study the sample efficiency of baselines and KIAN under the intra-task setup , where an agent learns a single task with the external knowledge set G init fixed. Figure 3 plots the learning curves in different environments. All experiments in these figures are run with ten random seeds, and each error band is a 95% confidence interval. The results of BC show that the external knowledge policies are sub-optimal for all environments. Given sub-optimal external knowledge, only KIAN shows success in all environments. In general, improvement of KIAN over baselines is more apparent when the task is more complex, e.g., Empty < Unlock < DoorKey and Push < Pick-and-Place. Moreover, KIAN is more stable than baselines in most environments. Note that in continuous-control tasks (Push, Slide, and Pick-and-Place), A2T barely succeeds since it does not consider the entropy imbalance issue introduced in Proposition 4.2 and 4.3. These results suggest that KIAN can more efficiently explore the environment with external knowledge policies and fuse multiple policies to solve a task.

Next, we evaluate the generalizability of all methods under simple-to-complex (S2C) and complex-tosimple (C2S) setups, where the former trains a policy in a simple task and test it in a complex one,

0.0

0.5

1.0

Timesteps

1e6

0.0

0.5

Reward

Dynamic-Obstacles-16x16

-> LavaCrossingS9N2

0

1

2

3

Timesteps

1e5

0.00

0.25

0.50

Reward

Dynamic-Obstacles-16x16

-> MultiRoom-N4-S5

0.0

0.5

1.0

Timesteps

1e6

0.0

0.5

Reward

Dynamic-Obstacles-16x16

-> KeyCorridorS3R2

0

2

4

Timesteps

1e5

0.0

0.5

1.0

Reward

Push -> Pick-and-Place

0

2

4

Timesteps

1e5

0.0

0.5

1.0

Reward

Unlock

-> DoorKey-8x8

0

1

2

3

Timesteps

1e5

0.00

0.25

0.50

Reward

DoorKey-8x8

-> MultiRoom-N4-S5

0.0

0.5

1.0

Timesteps

1e6

0.0

0.5

Reward

DoorKey-8x8

-> KeyCorridorS3R2

0

2

4

Timesteps

1e5

0.0

0.5

1.0

Reward

Slide -> Push

KIAN (for reference)

Cont. RL

Cont. KoGuN

Cont. A2T

Cont. KIAN

and the latter goes the opposite way. All generalizability experiments are run with the same policies as in Section 5.1. Table 1 and 2 show that KIAN outperforms other baselines in most experiments, and its results have a smaller variance (see Table 3 to 5 in Appendix E). These results demonstrate that KIAN's flexibility in incorporating external policies improves generalizability.

5.2 Compositional and Incremental Learning

In the final experiments, we test different methods in the compositional and incremental learning setting. We modify RL, KoGuN, and A2T to fit into this setting; details can be found in Appendix C. The experiments follow the inter-task setup : (1) We randomly select a pair of tasks ( M 1 k , M 2 k ) . (2) An agent learns a policy to solve M 1 k with G init fixed, as done in Section 5.1. (3) The learned (internal) policy, π 1 in , is added to the external knowledge set, G = G init ∪ { π 1 in } . (4) The same agent learns a policy to solve M 2 k with G . Each experiment is run with ten random seeds.

The learning curves in Figure 4 demonstrate that given the same updated G , KIAN requires fewer samples to solve M 2 k than RL, KoGuN, and A2T in all experiments. Our knowledge-key and query design disentangles policy representations from the action-prediction operation, so the agent is more optimized in incremental learning. Unlike our disentangled design, prior methods use a single function approximator to directly predict an action (KoGuN) or the weight of each policy (A2T) given a state. These methods make the action-prediction operation depend on the number of knowledge policies, so changing the size of G requires significant retraining of the entire function approximator.

Figure 4 also shows that KIAN solves M 2 k more efficiently with G than G init in most experiments. This improvement can be attributed to KIAN reusing the knowledge keys and query, which allows an agent to know which policies to fuse under different scenarios. Note that G can be further expanded with the internal policy learned in M 2 k and be used to solve another task M 3 k .

PPO+KIAN

PPO+KIAN(Entropy Imbalance)

Empty-Random-5x5

Unlock

DoorKey-5x5

DoorKey-8x8

1

1

1

Reward

Reward

Reward

Reward

0.5

0

0

0.0

0

0.0

2.5

5.0

7.5

0

1

2

3

0.0

2.5

5.0

7.5

0

1

2

3

Timesteps

1e4

Timesteps

1e5

Timesteps

1e4

Timesteps

1e6

Dynamic-Obstacles-16x16

LavaCrossingS9N2

MultiRoom-N4-S5

KeyCorridorS3R2

1

Reward

Reward

Reward

0

0.5

0.5

-1

0.0

0.0

0.0

0

1

2

3

0

2

4

0

2

4

0.0

0.5

1.0

Timesteps

1e5

Timesteps

1e6

Timesteps

1e5

Timesteps

1e6

SAC+KIAN

SAC+KIAN (Entropy Imbalance)

Pick-and-Place

Push

Slide

1

1

Reward

Reward

Reward

0.5

0

0

0.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

Timesteps

1e6

Timesteps

1e6

Timesteps

1e6

5.3 Analysis of Entropy Imbalance in Maximum Entropy KGRL

In our ablation study, we investigate (1) the impact of entropy imbalance on the performance of maximum entropy KGRL and (2) whether the proposed modifications to external policy distributions in Section 4.4 can alleviate the issue.

Figure 5 shows the learning curves comparing KIAN's performance with and without addressing the entropy-imbalance issue. The results demonstrate that when not addressing the issue using equation (5) or (8), KIAN fails to fully capitalize on the guidance offered by external policies. We also draw two noteworthy conclusions from the figure: (1) For discrete decision-making tasks, the detrimental impact of entropy imbalance becomes more evident as task complexity increases. (2) For continuous-control tasks, entropy imbalance can degrade KIAN's performance and make it perform worse than pure RL without external policies, as shown by the results of FetchPickAndPlace and FetchPush. This phenomenon can be attributed to Proposition 4.3. In contrast, by adjusting KIAN's external policy distributions using equation (5) or (8), a KGRL agent can efficiently harness external policies to solve a given task.

6 Conclusion and Discussion

This work introduces KGRL, an RL paradigm aiming to enhance efficient and flexible learning by harnessing external policies. We propose KIAN as an actor model for KGRL, which predicts an action by fusing multiple policies with an embedding-based attention operation. Furthermore, we propose modifications to KIAN's policy distributions to address entropy imbalance, which hinders efficient exploration with external policies in maximum entropy KGRL. Our experimental findings demonstrate that KIAN outperforms alternative methods incorporating external policies regarding sample efficiency, generalizability, and compositional and incremental learning.

However, it is essential to acknowledge a limitation not addressed in this work. The efficiency of KIAN, as well as other existing KGRL methods, may decrease when dealing with a large external knowledge set containing irrelevant policies. This issue is examined and discussed in Appendix F. Efficiently handling extensive sets of external policies is left for future research.

Our research represents an initial step towards the overarching goal of KGRL: learning a knowledge set with a diverse range of policies. These knowledge policies can be shared across various environments and continuously expanded, allowing artificial agents to flexibly query and learn from them. We provide detailed discussions on the broader impact of this work and outline potential directions of future research in Appendix D.

References

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. Advances in Neural Information Processing Systems , 35:28955-28971, 2022.

Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017.

Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15 , 1995.

A. Bandura. Social Learning Theory . Prentice-Hall series in social learning theory. Prentice Hall, 1977. ISBN 9780138167516.

Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid , 2018.

Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems , 5, 1992.

Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature , 602(7897):414-419, 2022.

Vinicius G Goecks, Gregory M Gremillion, Vernon J Lawhern, John Valasek, and Nicholas R Waytowich. Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments. arXiv preprint arXiv:1910.04281 , 2019.

Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning , pages 1352-1361. PMLR, 2017.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML , 2018.

Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John Agapiou, et al. Learning from demonstrations for real world reinforcement learning. 2017.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 , 2016.

Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems , 32, 2019.

Leslie Pack Kaelbling. The foundation of efficient robot learning. Science , 369(6506):915-916, 2020.

Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning , pages 651-673. PMLR, 2018.

Khimya Khetarpal and Doina Precup. Learning options with interest functions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 9955-9956, 2019.

Jaekyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. arXiv preprint arXiv:2106.14305 , 2021.

Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems , 29, 2016.

David JC MacKay. Information theory, inference and learning algorithms . Cambridge university press, 2003.

Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv:1810.01257 , 2018.

Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 6292-6299. IEEE, 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Mark Pfeiffer, Samarth Shukla, Matteo Turchetta, Cesar Cadena, Andreas Krause, Roland Siegwart, and Juan Nieto. Reinforced imitation: Sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations. IEEE Robotics and Automation Letters , 3(4):4423-4430, 2018.

Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464 , 2018.

Ahmed H. Qureshi, Jacob J. Johnson, Yuzhe Qin, Taylor Henderson, Byron Boots, and Michael C. Yip. Composing task-agnostic policies with deep reinforcement learning. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=H1ezFREtwH .

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021. URL http://jmlr.org/papers/v22/20-1364.html .

Janarthanan Rajendran, Aravind Lakshminarayanan, Mitesh M. Khapra, Prasanna P, and Balaraman Ravindran. Attend, adapt and transfer: Attentive deep architecture for adaptive transfer from multiple sources in the same domain. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=Sy6iJDqlx .

Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087 , 2017.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Seungmoon Song, Łukasz Kidzi´ nski, Xue Bin Peng, Carmichael Ong, Jennifer Hicks, Sergey Levine, Christopher G Atkeson, and Scott L Delp. Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation. Journal of neuroengineering and rehabilitation , 18(1):1-17, 2021.

Martin Stolle and Doina Precup. Learning options in reinforcement learning. In International Symposium on abstraction, reformulation, and approximation , pages 212-223. Springer, 2002.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Wei-Cheng Tseng, Jin-Siang Lin, Yao-Min Feng, and Min Sun. Toward robust long range policy transfer. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9958-9966, 2021.

Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 , 2017.

Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature , 602(7896):223-228, 2022.

Peng Zhang, Jianye Hao, Weixun Wang, Hongyao Tang, Yi Ma, Yihai Duan, and Yan Zheng. Kogun: Accelerating deep reinforcement learning via integrating human suboptimal knowledge. In International Joint Conference on Artificial Intelligence , 2020.

Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy . Carnegie Mellon University, 2010.

A Proofs and Learning Algorithms of KIAN

A.1 Proof of Proposition 4.1

Proposition 4.1 (Entropy imbalance in discrete decision-making) . Assume that a d a -dimensional probability simplex π ∈ ∆ d a is fused by { π 1 , . . . , π m } and { ˆ w 1 , . . . , ˆ w m } following equation (3), where π j ∈ ∆ d a , ˆ w j ≥ 0 ∀ j ∈ { 1 , . . . , m } and ∑ m j =1 ˆ w j = 1 . If the entropy of π is maximized and ∥ π 1 ∥ ∞ ≪ ∥ π 2 ∥ ∞ , ∥ π 1 ∥ ∞ ≪ ∥ π 3 ∥ ∞ , . . . , ∥ π 1 ∥ ∞ ≪∥ π m ∥ ∞ , then ˆ w 1 → 1 .

Proof. Since π ∈ ∆ d a is defined as equation (3), and its entropy is maximized,

Equation (12) holds since π becomes a uniform distribution if its entropy is maximized.

Since each π j in equation (12) is a probability simplex, it can be written as

Substituting equation (13) into equation (12) and, after some rearrangement, we get

Without loss of generality, we can assume that

Then take the infinity norm for both sides of equation (15), we get

After some rearrangement, equation (19) becomes

Equation (21) holds since ∑ m j =1 ˆ w j = 1 .

Given equation (14), (17), and the assumption that ∥ π 1 ∥ ∞ ≪ ∥ π 2 ∥ ∞ , ∥ π 1 ∥ ∞ ≪ ∥ π 3 ∥ ∞ , . . . , ∥ π 1 ∥ ∞ ≪ ∥ π m ∥ ∞ , we have the following inequalities

Hence, for equation (21) to hold, ˆ w 1 → 1 .

Proposition 4.1 states that if ∥ π 1 ∥ ∞ is much smaller than ∥ π j ∥ ∞ , then maximizing the entropy of π results in ˆ w 1 → 1 and ˆ w j → 0 , ∀ j ∈ { 2 , . . . , m } . Next, we provide another proposition showing that if π 1 is a uniform distribution and π j is not, then ∥ π 1 ∥ ∞ < ∥ π j ∥ ∞ , ∀ j ∈ { 2 , . . . , m } .

Proposition A.1 (Infinity norm of a probability simplex) . Given two d a -dimensional probability simplices, π 1 ∈ ∆ d a and π 2 ∈ ∆ d a , if π 1 is a uniform distribution and π 2 is not, then ∥ π 1 ∥ ∞ = 1 d a < ∥ π 2 ∥ ∞ .

Proof. Since π 1 ∈ ∆ d a is a uniform distribution, π 1 = [ 1 d a , 1 d a , . . . , 1 d a ] ⊤ . The infinity norm of π 1 becomes

where π 1 ,i is the i -th element of π 1 . On the other hand, since π 2 ∈ ∆ d a is not a uniform distribution, it can be represented as

̸

Equation (25) indicates that at least one element in { ε 2 , 1 , . . . , ε 2 ,d a } should be larger than 0 . Hence,

A.2 Proof of Proposition 4.2

In continuous action space, the final policy fused by equation (3) is a mixture of normal distributions, also known as a Gaussian mixture. However, in general, the entropy of a Gaussian mixture does not have a closed form [45]. Instead of analyzing the entropy of π , we analyze the variance of π for d a = 1 in maximum entropy KGRL since for any probability density function of a real-valued random variable, Shannon's inequality for entropy and variance [44, 47] specifies

where σ 2 ∈ R ≥ 0 is the variance of π and p ≈ 3 . 14159 .

Proposition 4.2 (Entropy imbalance in continuous control) . Assume a one-dimensional policy distribution π is fused by

If the variance of π is maximized, and σ 2 1 ≫ σ 2 2 and σ 2 1 ≫ ( µ 1 -µ 2 ) 2 , then ˆ w 1 → 1 .

Proof. Let a be a continuous random variable with the probability density function being π ( ·| s ) . Then its first and second moments are

Equation (36) holds since σ 2 j = E a j ∼ π j [ a 2 j ] -( E a j ∼ π j [ a j ] ) 2 , ∀ j ∈ { 1 , 2 } . The variance of a thus becomes

According to equation (41), if σ 2 1 ≫ σ 2 2 and σ 2 1 ≫ ( µ 1 -µ 2 ) 2 , maximizing V a ∼ π [ a ] leads to ˆ w 1 → 1 .

A.3 Proof of Proposition 4.3

Proposition 4.3 (Distribution separation in continuous control) . Assume a one-dimensional policy distribution π is fused by equation (4). If ˆ w 1 , ˆ w 2 , σ 2 1 , and σ 2 2 are fixed, then maximizing the variance of π will increase the distance between µ 1 and µ 2 .

Proof. According to equation (41), if ˆ w 1 , ˆ w 2 , σ 2 1 , and σ 2 2 are fixed, maximizing V a ∼ π [ a ] results in maximizing ( µ 1 -µ 2 ) 2 , hence increasing the distance between µ 1 and µ 2 .

A.4 Proof of Proposition 4.4

Before proving Proposition 4.4, we first show that KL divergence between a mixture of uniform distributions and a mixture of normal distributions is upper-bounded by a constant.

Proposition A.2 (KL divergence between a mixture of uniform distributions and a Gaussian mixture) . Given a Gaussian mixture

and a mixture of m uniform distributions

for a real-valued random variable, the KL divergence between ˆ π ( · ) and π ( · ) has an upper bound of p 12 .

Proof. Since KL divergence D KL ( P ∥ Q ) is convex in the pair ( P, Q ) , D KL (ˆ π ∥ π ) has the following upper bound [43]

For each j ∈ { 1 , . . . , m } ,

Equation (53) comes from substituting γ j into equation (52).

Finally, the upper bound of D KL (ˆ π ∥ π ) becomes

Proposition 4.4 (Approximation of a mixture of normal distributions) . If the following three inequalities hold for µ t,in , µ t,g 1 , . . . , µ t,g n , and a t,in : ∥ µ t,in -µ t,g j ∥ 2 < min { γ t,in , γ t,g j } , ∥ a t,in -µ t,in ∥ 2 < min { γ t,in , γ t,g j } , and ∥ a t,in -µ t,g j ∥ 2 < γ t,g j , ∀ j ∈ { 1 , . . . , n } , where γ t,in = 1 / (2 π in ( µ t,in | s t )) and γ t,g j = 1 / (2 π g j ( µ t,g j | s t )) , then equation (9) for a real-valued action a t sampled from KIAN can be approximated by

In addition, equation (8) is a lower bound of equation (10).

Proof. Proposition A.2 shows that π ( ·| s t ) fused as equation (3) can be approximated by

with KL divergence being at most p 12 , which is a constant.

Since any continuous action a t outputted by KIAN belongs to { a t,in , µ t,g 1 , . . . , µ t,g n } (Line 11 to 16 in Algorithm 1), if the three inequalities in the proposition statement hold, then for all a t

Therefore, ˆ π ( ·| s t ) can be rewritten as

For any a t ∈ { a t,in , µ t,g 1 , . . . , µ t,g n } ,

Inequality (66) holds since π in ( a t | s t ) has a maximum value when a t = µ t,in , and it shows that equation (8) is a lower bound of equation (10).

We use equation (8) instead of (10) to approximated equation (9) since it includes more information about the distribution π in ( ·| s t ) and helps adjust the learnable variance σ 2 t,in .

A.5 Proof of Proposition 4.5

Proposition 4.5 (Maximized variance's independence of the distance between means) . Assume a onedimensional policy π is fused by equation (4). If π ( a | s ) is approximated as equation (8), and the three inequalities in Proposition 4.4 are satisfied, then maximizing the variance of π ( ·| s ) will not affect the distance between µ 1 and µ 2 .

Proof. Proposition 4.4 shows that approximating π ( a | s ) as equation (8) comes from approximating π ( ·| s ) with

Let a be a continuous random variable with the probability density function being ˆ π ( ·| s ) . Following the proof of Proposition 4.2, its first and second moments are

Equation (69) holds since the variance of U ( · ; a, b ) is ( b -a ) 2 12 . Then the variance of a becomes

which is not related to the distance between µ 1 and µ 2 .

A.6 Learning Algorithms for KIAN

Algorithm 1: Knowledge-Grounded RL with KIAN

B Experimental Details

All experiments are conducted using Pytorch [22].

B.1 Baseline Algorithms

We compare KIAN with the following baselines that incorporate external knowledge policies differently.

Behavior cloning (BC) [3]: An agent follows only the policies in G to solve a task. This method is the optimal solution for supervised learning from demonstrations, where external knowledge policies generate the demonstrations.

RL [10, 29]: An agent learns a policy by RL without any external guidance.

RL+BC [21]: An agent learns a policy by RL with BC signals integrated. These signals come from demonstrations generated by external knowledge policies.

KoGuN [36]: An agent learns a policy with the input being a concatenation of a state and n actions suggested by all policies in G .

A2T [27]: An agent fuses a learnable inner policy with n external policies in G by learning a function approximator that predicts n +1 weights for all policies.

B.2 MiniGrid Environments

B.2.1 Environmental Details

We evaluate all methods on the following tasks in MiniGrid environments ( https://github.com/maximecb/ gym-minigrid ): Empty-Random-5x5, Unlock, DoorKey-5x5, DoorKey-8x8, Dynamic-Obstacles-16x16, LavaCrossingS9N2, MultiRoom-N4-S5, and KeyCorridorS3R2. A state s t in each task is a directed firstperson view represented as a 5x5 grid. An action a t in each task is one of the six discrete actions: left, right, forward, pickup, drop, and toggle.

B.2.2 Initial External Knowledge Set

The initial external knowledge set, G init , for MiniGrid tasks comprises eight sub-optimal if-else-based programs, such as:

pick_up_the_key : If there exists a key in s t , move to the key; if the key is in front of the agent, π g j ( pickup | s t ) = 1 .

pick_up_the_ball : If there exists a ball in s t , move to the key; if the key is in front of the agent, π g j ( pickup | s t ) = 1 .

open_the_door : If there exists a door in s t , move to the door; if the door is in front of the agent, π g j ( toggle | s t ) = 1 .

open_the_locked_door : If there exists a locked door in s t , move to the door; if the door is in front of the agent, π g j ( toggle | s t ) = 1 .

open_the_unlocked_door : If there exists a unlocked door in s t , move to the door; if the door is in front of the agent, π g j ( toggle | s t ) = 1 .

go_to_the_goal : If there exists a goal in s t , move to the goal.

do_not_hit : If there exists walls or lava around the agent, do not choose the direction.

do_not_hit_balls : If there exists balls around the agent, do not choose the direction.

In the above policies, the 'move to' is decided by p obj -p self , where p self ∈ R 3 is the position of the agent and p obj ∈ R 3 is the position of the object. The π g j of the actions right, left, and forward, can be written as:

B.2.3 Model Architecture

The eight environments in Minigrid share the same model architecture. Each method involves learning an image encoding, an actor and a critic networks. The architecture of the image encoding network and critic network are the same for all methods, but their actor networks have different architectures.

Image Encoding Network. The image encoding network is a three-layer convolutional neural network that maps an input image to an image embedding, which is used as the state by actor and critic networks.

Critic Network. A critic network is a multi-layer perceptron (MLP) that predicts a state value [29]. The architecture of a critic network has one hidden layer that contains 64 units. Each hidden layer is followed by Tanh activation.

Actor Network of PPO, PPO+BC, and KoGuN. An actor network of PPO, PPO+BC and KoGuN is an MLP with one hidden layers and a hidden size of 64 units.

Actor Network of A2T. An actor network of A2T contains an internal actor network and an attention network. The internal actor network has the same architecture as an actor of PPO, PPO+BC, and KoGuN. The attention network is an MLP with one hidden layer and a hidden size of 64 units base.

KIAN. The internal actor network of KIAN has the same architecture as an actor of PPO, PPO+BC, and KoGuN. Each knowledge key is a learnable vector with d k = 8 and modeled by the PyTorch module, nn.Embedding . The query network is an additional output layer projecting 64-dim to d k -dim that based on the inner actor's first layer outputs.

B.2.4 Hyperparameters

We implement all methods based on the implementation of PPO in https://github.com/lcswillems/ rl-starter-files . The training timesteps are 75K, 300K, 75K, 3M, 300K, 5M, 500K, 1M for Empty-Random-5x5, Unlock, DoorKey-5x5, DoorKey-8x8, Dynamic-Obstacles-16x16, LavaCrossingS9N2, MultiRoom-N4-S5, and KeyCorridorS3R2 respectively. The learning rates are 1 × 10 -3 for all tasks. The batch sizes are 256 for all tasks. The discount factors γ = 0 . 99 for all tasks. The coefficient of the entropy term α is searched to be 0 or 0.01, the default value.

B.3 OpenAI-Robotic Environments

B.3.1 Environmental Details

We evaluate all methods on the following tasks in OpenAI-Robotic environments: FetchPush, FetchSlide, and FetchPickAndPlace. A state s t ∈ R 25 in each task contains (1) the position and velocity of the end-effector, (2) the position, rotation, and velocity of the object, (3) the relative position between the object and the end-effector, and (4) the distance between the two grippers and their velocity. An action a t ∈ R 4 in each task contains the position variation of the end-effector and the distance between the two grippers.

B.3.2 Initial External Knowledge Set

The initial external knowledge set, G init , for all OpenAI-Robotic tasks comprises two sub-optimal if-else-based programs, move_forward_to_the_object and move_forward_to_the_goal .

move_forward_to_the_object : If ∥ p ee -p obj ∥ 2 ≥ ε , move straightly to the object with the gripper opened; otherwise, stay unmoved.

move_forward_to_the_goal : If ∥ p ee -p obj ∥ 2 < ε , move straightly to the goal with the gripper closed; otherwise, stay unmoved.

In the above two policies, p ee ∈ R 3 is the position of the end-effector, and p obj ∈ R 3 is the position of the object. For all tasks, ε = 0 . 03 .

B.3.3 Model Architecture

FetchPush, FetchSlide, and FetchPickAndPlace share the same model architecture. Each method involves learning an actor and a critic network. The architecture of the critic network is the same for all methods, but their actor networks have different architectures.

Critic Network. A critic network is a multi-layer perceptron (MLP) that predicts a state-action value [ ? ]. The architecture of a critic network has three hidden layers, and each layer contains 512 units. Each hidden layer is followed by ReLU activation.

Actor Network of SAC, SAC+BC, and KoGuN. An actor network of SAC, SAC+BC and KoGuN is an MLP with three hidden layers and a hidden size of 512 units.

Actor Network of A2T. An actor network of A2T contains an internal actor network and an attention network. The internal actor network has the same architecture as an actor of SAC, SAC+BC, and KoGuN. The attention network is an MLP with two hidden layers and a hidden size of 64 units.

KIAN. The internal actor network of KIAN has the same architecture as an actor of SAC, SAC+BC, and KoGuN. Each knowledge key is a learnable vector with d k = 4 and modeled by the PyTorch module, nn.Embedding . The query network is an MLP with two hidden layers and a hidden size of 64 units.

B.3.4 Hyperparameters

We implement all methods based on the implementation of SAC in Stable-Baselines3 (SB3) [26]. The training timesteps are 1M for FetchPush and FetchPickAndPlace and 1.2M for FetchSlide. The learning rates are 5 × 10 -4 for FetchPush and FetchPickAndPlace and 4 × 10 -4 for FetchSlide. The batch sizes are 2048 for all tasks. The replay-buffer sizes are 1M for all tasks. The discount factors γ = 0 . 95 for all tasks. The coefficient of the entropy term α is adjusted automatically for all tasks as described in [10].

C Details of Compositional and Incremental Experiments

C.1 MiniGrid Environments

After learning an actor in M 1 k with the experimental setup described in Section B, we train KIAN for M 2 k by initializing its external knowledge keys with the knowledge keys learned in M 1 k . These external knowledge keys remain fixed when learning M 2 k , and all other components of KIAN are learned from scratch. This setup allows us to test the efficacy of reusing learned knowledge keys across different tasks. All actor components of RL, KoGuN, and A2T are learned from scratch in M 2 k . This is because RL does not incorporate any external knowledge, and the model architectures of KoGuN and A2T do not allow changing the number and order of knowledge policies. The hyperparameters of learning M 1 k and M 2 k are the same as those listed in Section B.2.4. The external knowledge policies used for each experiment are detailed as follows:

Dynamic-Obstacles-16x16 → LavaCrossingS9N2. Reuse the knowledge embedding of 'go to the goal'; reuse the knowledge embedding of 'do not hit balls' as the fixed knowledge embedding of 'do not hit'.

Unlock → DoorKey-8x8. Reuse the knowledge embeddings of 'get the key' and 'open the door'.

Dynamic-Obstacles-16x16 → MultiRoom-N4-S5. Reuse the knowledge embedding of 'go to the goal'.

DoorKey-8x8 → MultiRoom-N4-S5. Reuse the knowledge embedding of 'go to the goal'; reuse the knowledge embedding of 'open the door' as the fixed knowledge embedding of 'open the unlocked door'.

Dynamic-Obstacles-16x16 → KeyCorridorS3R2. Reuse the knowledge embedding of 'go to the goal' as the fixed knowledge embedding of 'pick up the ball'.

DoorKey-8x8 → KeyCorridorS3R2. Reuse the knowledge embedding of 'get the key'; reuse the knowledge embedding of 'open the door'; reuse the knowledge embedding of 'go to the goal' as the fixed knowledge embedding of 'pick up the ball'.

C.2 OpenAI-Robotic Environments

After learning an actor and a critic in M 1 k with the experimental setup described in Section B, we initialize the networks for M 2 k as follows:

RL : The actor and critic of M 2 k are initialized with that of M 1 k . These networks will be updated when learning in M 2 k .

KoGuN : Only the critic of M 2 k is initialized with that of M 1 k . The actor is learned from scratch in M 2 k . The actor and critic will be updated when learning in M 2 k .

A2T : The critic of M 2 k is initialized with that of M 1 k . The inner policy and attention network are learned from scratch in M 2 k . All networks will be updated when learning in M 2 k .

KIAN : The external knowledge keys, query, and critic of M 2 k are initialized with that of M 1 k . Note that the external knowledge keys of M 2 k include the internal and external knowledge keys from M 1 k . The inner policy and inner knowledge key are learned from scratch in M 2 k . The external knowledge keys remain fixed, while other networks will be updated when learning in M 2 k .

The hyperparameters of learning M 1 k are the same as those listed in Section B.3.4. When learning M 2 k , the hyperparameters changed are listed as follows: The training timesteps are 0.4M. The learning rates are 7 . 5 × 10 -4 and 10 -3 for FetchPush and FetchPickAndPlace respectively.

D Broader Impact and Future Research Directions

The KGRL framework presented in this work aims to enhance an agent's ability to learn from external policies. These policies encompass not only sub-optimal strategies to task completion but also regulative policies that emphasize safety constraints and ethical behaviors. Being able to incorporate safety- and ethics-oriented policies gives KGRL the potential to significantly influence an artificial agent's behavior, promoting enhanced safety and social acceptability. These aspects have gained substantial attention in the field of RL [38-42, 46, 48], underscoring their importance in contemporary research.

Moving forward, there are several research directions in KGRL that are worth exploring. First, fusing knowledge policies with different state and action spaces enables efficient learning across a broader range of applications. Second, integrating regulative policies that enforce strict constraints during learning and inference stages can ensure adherence to safety and ethical considerations. Lastly, addressing complex relationships among different policies, such as conditional dependence and conflicts, allows an agent to efficiently navigate through a large and diverse set of knowledge policies. We hope these directions have the potential to inspire future studies in KGRL.

E Other Experimental Results

We provide the standard deviation for the experiments of generalizability in Table 3-5.

F Effects of Extensive External Knowledge Set with Irrelevant Policies

In order for KGRL agents to effectively leverage an external knowledge set, it is imperative that they can (1) distinguish which external policies are less related to the given task and (2) efficiently navigate through an

extensive collection of external policies. Failure to accomplish these objectives in a timely manner could result in suboptimal performance, potentially even inferior to that of an RL agent. Under such circumstances, integrating external policies into the learning process becomes impractical.

In this section, we examine the following two aspects of KIAN: (1) the ability of KIAN to rapidly acquire valuable strategies, even in the presence of random or irrelevant policies within the external knowledge set, and (2) the impact of KIAN's performance as the size of the external knowledge set increases.

Figure 6 shows the learning curves of PPO and PPO+KIAN, considering various numbers of external policies: 2 relevant, 4 (2 relevant + 2 irrelevant), and 6 (2 relevant + 4 irrelevant) for the MiniGrid Unlock task. The results indicate that including more irrelevant knowledge policies leads to a marginal decline in performance, but the agents consistently achieve high rewards with minimal variances. This minor decline in performance aligns with our expectations since the agents need to (1) navigate through a more extensive set of external policies and (2) distinguish and disregard policies that do not contribute to solving the task. Therefore, when the external knowledge set is very large, KGRL methods, such as KoGuN, A2T, and KIAN, do not guarantee superior efficiency over RL methods. Efficiently harnessing a large external policy set remains an avenue of future research and exploration.

0

1

2

3

Timesteps

1e5

0.0

0.5

1.0

Reward

Unlock

PPO

PPO+KIAN

PPO+KIAN+Redundant2x

PPO+KIAN+Redundant3x

Reference

Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 3387-3395, 2019.

Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. Advances in neural information processing systems , 31, 2018.

Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. In International Conference on Artificial Intelligence and Statistics , pages 3304-3312. PMLR, 2021.

Adrien Ecoffet and Joel Lehman. Reinforcement learning under moral uncertainty. In International conference on machine learning , pages 2926-2936. PMLR, 2021.

Javier Garcia and Fernando Fernández. Safe exploration of state and action spaces in reinforcement learning. Journal of Artificial Intelligence Research , 45:515-564, 2012.

John R Hershey and Peder A Olsen. Approximating the kullback leibler divergence between gaussian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal ProcessingICASSP'07 , volume 4, pages IV-317. IEEE, 2007.

Isidore I Hirschman. A note on entropy. American journal of mathematics , 79(1):152-156, 1957.

Marco F Huber, Tim Bailey, Hugh Durrant-Whyte, and Uwe D Hanebeck. On entropy approximation for gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems , pages 181-188. IEEE, 2008.

Puze Liu, Davide Tateo, Haitham Bou Ammar, and Jan Peters. Robot reinforcement learning on the constraint manifold. In Conference on Robot Learning , pages 1357-1366. PMLR, 2022.

Claude E Shannon. A mathematical theory of communication. The Bell system technical journal , 27(3): 379-423, 1948.

Yueh-Hua Wu and Shou-De Lin. A low-cost ethics shaping approach for designing reinforcement learning agents. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces KIAN, a method for leveraging multiple sub-optimal policies to solve reinforcement learning tasks. The paper starts out by introducing a new Knowledge-Grounded MDP, which adds a set of external knowledge policies to the traditional MDP framework. \n\nTo leverage these external knowledge policies, KIAN learns an embedding for each knowledge policy (including an inner knowledge policy for the current agent). Then at each step, the current state is mapped to a query embedding that can be used to find the policy that is best equipped to take an action at that timestep.\n\nSTRENGTHS:\n* The paper is structured well and is easy to read.\n* The authors demonstrated great attention to detail by motivating the theorems and equations with intuition before defining them concretely.\n*  The paper is very technically sound.\n\nWEAKNESSES:\n* > In such a case, whenever the knowledge set is updated by adding or replacing policies, prior methods require relearning the entire multi-policy fusion process, even if the current task is similar to the previous one. This is because their designs of knowledge representations are intertwined with the knowledge-fusing mechanism, which restricts the number of policies in the knowledge set from being changed.\n\n    * It would be good to point to specific prior works that suffer from this problem so the reader can build intuition.\n\n* There are no visualizations of the tasks used for evaluation. Adding pictures of the environments would help readers understand what the agent needs to do.\n\nQUESTIONS:\n*  I don't fully understand the incremental property. In the introduction. the authors state that \"Humans do not need to relearn how to navigate the entire knowledge set from scratch when they remove outdated strategies or add new ones.\" But then Definition 3.5 states that an incremental agent has two properties:\n    1. The agent can use different knowledge policies to solve a single KGMDP\n    2. Given a sequence of KGMDPs, the agent can solve them with different knowledge sets.\n\n    * In its current form, definition 3.5 does not specify the case where an agent adds or removes a specific policy from its knowledge set. I think a condition should be added to highlight how an incremental agent should be able to leverage a sequence of knowledge sets that differ by one or more policies.\n\n\n* How do the initial external knowledge policies perform on the given tasks?\n\nLIMITATIONS:\nThe authors discussed broader impact and future work in Appendix D, but no limitations were addressed. Could the authors please add limitations to either Appendix D or the conclusion?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this work introduce Knowledge-Grounded RL (KGRL), an RL framework that combines multiple knowledge policies to achieve human-like efficiency and flexibility in learning. They propose a new actor architecture called Knowledge-Inclusive Attention Network (KIAN) that enables arbitrary combinations and replacements of external policies through embedding-based attentive action prediction. KIAN also addresses entropy imbalance, a challenge in KGRL, addressing exploration in the environment.\n\nSTRENGTHS:\nFirstly, the paper addresses a relevant and interesting problem in the field of reinforcement learning (RL) by improving sample efficiency from arbitrary external policies and enabling knowledge transfer skills. \nFurthermore, the paper is a well-written work that effectively conveys the ideas and findings in a clear and concise manner. The authors demonstrate excellent writing skills, employing appropriate terminology and organizing the content in a manner that enhances readability and comprehensibility.\nIn addition, the paper effectively establishes the motivation and position of the study in the existing literature. The authors articulate the significance and relevance of the research problem, demonstrating a strong understanding of the field. They situate their work within the broader scholarly context, highlighting how their study fills a gap in knowledge or builds upon prior research.\nThe methodology employed in the research is clearly described, allowing readers to understand and replicate the study. The authors provide a detailed and comprehensive explanation of the experimental and theoretical approach used, supported by well-designed figures and diagrams. These visual aids enhance the understanding of the methods employed and facilitate better comprehension of the research. Lastly, the authors provide solid mathematical understanding for their proposed method. \n\nCollectively, these strengths highlight the quality of the paper. ts focus on addressing a relevant and interesting problem, combined with its well-written content, clear methodology, positioning in the literature, and mathematical rigor, make it an intersting contribution.\n\nWEAKNESSES:\nThe paper has one notable weakness that should be addressed to enhance the overall quality. The evaluation presented in the paper is quite limited. The results obtained for the OpenAI robotics task only offer a slight support for the proposed method. The authors should consider expanding the evaluation to provide a more comprehensive assessment of the proposed method's performance.\nAdditionally, the use of only 3 or 5 seeds in the experiments may be insufficient, especially when considering the large error bars observed in some models. The authors should consider increasing the number of seeds to improve the statistical robustness of the results.\nMoreover, the paper lacks clarity regarding the error bars shown in the plots. It is not explicitly stated what these error bars represent, which hinders the interpretation and understanding of the presented data.\nLastly, the paper would greatly benefit from conducting ablation studies to investigate the effects of various factors on the proposed method's performance. Specifically, the authors could consider performing ablation studies on \n- the influence and distribution of attention/weights of actors, \n- the impact of random/irrelevant policies in $\\mathcal{G}$,\n- the impact of the (near) optimal policy in $\\mathcal{G}$, \n- the impact of a larger set of knowledge policies $\\mathcal{G}$, \n- the effects of different types of knowledge policies, \n- and an investigation of the mentioned entropy balance issue, that the authors specifically address in their method.\nThese ablation studies would provide valuable insights into the individual contributions and impacts of these factors on the overall approach.\n\nOverall, addressing these weaknesses would significantly improve the research paper, clarifying important aspects such as error bars, conducting relevant ablation studies, ensuring consistency in reporting variances, and strengthening the empirical evidence for the proposed method.\n\n### Minor\n- I assume ori-KIAN is KIAN with the original $\\mathcal{G}$, this should be mentioned in the text. \n- l.314 mentions less variance, but tables do not show variances for individual experiments\n\nQUESTIONS:\nSee the above mentioned ablation studies.\n\nLIMITATIONS:\nThe limitations have not been discussed by the authors.\nThe above mentioned ablation studies will probably help to identify the shortcomings of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nHumans can learn by aggregating external knowledge from others’ policies of attempting a task. While prior studies in RL have incorporated external knowledge policies for sample efficiency,  there still remains the generalization problem to be solved that agents have difficulties to perform arbitrary combinations and replacements of external policies.\nThe authors propose a new actor architecture for Knowledge-Grounded RL (KGRL),  Knowledge-Inclusive Attention Network (KIAN), which allows free knowledge rearrangement due to embedding-based attentive action prediction.  KIAN addresses entropy imbalance as well. The authors demonstrate in experiments that KIAN outperforms other methods incorporating external knowledge policies under different environmental setups.\n\nSTRENGTHS:\nThe authors clearly define the problem as how RL can be grounded on any given set of external knowledge policies to achieve knowledge-acquirable, sample-efficient, generalizable, compositional, and incremental properties.\nThe proposed method of KIAN is clearly described. They use knowledge keys and the query performs an attention operation to determine how an agent integrates all policies. \nThe solution of the entropy imbalance issues when integrating external policies are proposed as well.\n\nWEAKNESSES:\nGenerally this work has many related works but is tackling the unique challenge problem of fusing knowledge policies with different state and action spaces.\nLimitations of the proposed method is not clear based on the experimental results.\n\nQUESTIONS:\nMore experiments directly connected with this challenging problem would more clearly support your claims.\n\nLIMITATIONS:\nSome insights and analysis about limitations based on the experimental environments more directly connected with the target problem are expected to be included.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper defines the Knowledge-Grounded RL setting, a general RL setting for integrating knowledge (in the form of policies) into a policy to learn new tasks efficiently. Essentially this setting is similar to the MDP setting except that the agent is also given a set of knowledge policies to utilize. The paper also introduces a system/architecture within this KGRL setting, called Knowledge-Inclusive Attention Network (KIAN). The aim is to improve RL that is grounded on external knowledge policies. The paper outlines five desirable human-like properties they desire in their agents: knowledge-acquirable, sample-efficient, generalizable, compositional, and incremental. Moreover, they formally define these so that they are measurable within the KGRL setting (e.g., for evaluating algorithms on these dimensions).\n\nWhile previous methods typically intertwine knowledge representation and knowledge-fusion, thereby restricting their ability to adapt to numbers of policies, losing flexibility. KIAN is developed more flexibility, separating the knowledge representation and knowledge fusion.\n\nKIAN consists of three components: a policy that learns a strategy (similar to a normal policy) called the internal, embeddings that represent the given knowledge (or external) policies, a query that performs attentive action prediction to fuse the internal and external policies.\n\nKIAN also solves other issues that can occur in entropy-regularized KGRL. Entropy-regularized RL is common, but the authors show that in the KGRL setting issues can arise through entropy regularization where only a select few policies are selected, counterproductively reducing diversity in policy usage. The authors show that in the KGRL setting the agent will pay more attention to the policy with large entropy and in continuous control, will rely on the internal policy extensively. The paper introduces modifications so that this does not occur in KIAN.\n\nThe authors show results on both MiniGrid and robotics tasks and demonstrate sample efficiency, generalizability, as well as compositional and incremental learning.\n\nSTRENGTHS:\nThe paper is mostly well-written and well-explained.\n\nThe method makes sense, and is a well-thought out architecture. \n\nI like how the authors address the entropy imbalance problem.\n\nI like that the authors define and quantify the behaviors they would like in the agent.\n\nThe results do seem to demonstrate their method is effective.\n\nWEAKNESSES:\nWhile I think the experiments are good, with appropriate baselines and good environments to test the agent’s capabilities. I am concerned about statistical significance. In particular, only 5 seeds are run, and the performance benefit in many cases is minimal, which may quite possibly be attributed to randomness. While I do believe the method outperforms the baselines, I cannot say so with a lot of confidence to merit inclusion at NeurIPS. If the authors can run more seeds, especially given the large variance, it would dramatically improve their results.\n\n\nQualms:\nKGRL is consistently referred to as an RL framework, which it is, but the connotation can be misconstrued as being a framework “for” RL, implying it is a solution method for RL problems. I would recommend calling it a “setting” rather than a framework. Indeed, I was confused temporarily as a reader, especially when KGRL is stated as being “introduced” by this paper (as opposed to “described” or “outlined”). \n\n\nNits\nTypo Line 345: “border” should be: “broader”\n\nQUESTIONS:\nIn line 33, it is stated (for incremental): “Humans do not need to relearn how to navigate the entire knowledge set from scratch when they remove outdated strategies or add new ones”. Do humans truly “remove” outdated strategies?\n\nIn line 222, the paper states: “However, fusing multiple policies as equation (3) will make an agent biased toward a small set of knowledge policies when exploring the environment.” I am somewhat confused, equation (3) as is does not seem to have this issue. I thought this only occurs in the MaxEnt KGRL setting as introduced in the next section. Can the authors please clarify this?\n\nThe authors state the entropy imbalance problem is a property of the maximum entropy KGRL setting. I want to clarify that this is only shown for the specific case of policies in the form of equation (3), correct?\n\nHow could the policy embeddings/knowledge keys be learned?\n\nIs Gymnasium Robotics used for the experiments? Or the older OpenAI codebase?\n\nLIMITATIONS:\nTo me it is unclear how in more sophisticated settings how the knowledge keys (or policy embeddings) would be learned. This seems like a bottleneck to scalability that is not well-addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}