Knowledge-Grounded Reinforcement Learning
Authors: Zih-Yun Chiu
Yi-Lin Tuan
William Yang Wang
Michael C. Yip
Venue: Submitted to ICLR 2023 (2023)

Abstract: Receiving knowledge, abiding by laws, and being aware of regulations are common behaviors in human society. Bearing in mind that reinforcement learning (RL) algorithms benefit from mimicking humanity, in this work, we propose that an RL agent can act on external guidance in both its learning process and model deployment, making the agent more socially acceptable. We introduce the concept, Knowledge-Grounded RL (KGRL), with a formal definition that an agent learns to follow external guidelines and develop its own policy. Moving towards the goal of KGRL, we propose a novel actor model with an embedding-based attention mechanism that can attend to either a learnable internal policy or external knowledge. The proposed method is orthogonal to training algorithms, and the external knowledge can be flexibly recomposed, rearranged, and reused in both training and inference stages. Through experiments on tasks with discrete and continuous action space, our KGRL agent is shown to be more sample efficient and generalizable, and it has flexibly rearrangeable knowledge embeddings and interpretable behaviors.

Under review as a conference paper at ICLR 2023
KNOWLEDGE-GROUNDER T LEARN- ING
Anonymous authors
Paper under double-bling review
ABSTRACT
Receiving knowledge, abiding by laws, and being aware of regulations are common behaviors in human society. Bearing in mind that reinforcement learning (RL) algorithms benefit from mimicking humanity, in this work, we propose that an RI. agent can act on external guidance both its learning process and model deployment, making the agent more socially acceptable. We introduce the concept, Showledge-Groundee RL (KGRL), with a formal definition that an agent learns to follow external guidelines and develop its own policy. Moving towards the goal of KGRL, we propose a novel actor model with an ambedding-basee attention mecha- nism that attend to either learnable internal policy external knowledge. The proposed method is orthogonal to training algorithms, and the external knowledge can be flexibly recomposed, rearranged, and reused in both training and inference stages. Through experiments on tasks with discrete and continuous action space, our KGRL agent is shown to be more sample efficient and generalizable, and it has flexibly rearrangeable knowledge embeddings and interpretable behaviors.
INTRODUCTION
Incorporating external guidance into learning is a commonly seen behavior among humans. We can speed up our learning process by referring useful suggestions. At the same time, we beware of external regulations for safe and ethical reasons. On top of following external guidance, we humans can learn our own strategies to complete a task. We can also arbitrarily recompose, rearrange, and reuse those strategies and external guidelines to solve new task and adapt to environmental changes. Imitating human behaviors has been shown to benefit reinforcement learning (RI.) (Billard et al., 2016, (Sutton and Barto 2018 Zhang et al. 2019). However, how an RL agent can achieve the above capabilities remains challenging.
Different approaches have been proposed to develop some of these abilities. One branch of previous work in RL has studied how an agent can learn from demonstrations provided externally as examples of completing a task (Ross et al. [201T: Rajeswaran al. 2017 Duan et al. 2017 Nair et al.) 2018
Goecks et al. 2019 Ding et al. (219)) Another branch of previous research has investigated how an agent can learn reusable policies. This branch of research includes (1) transferring knowledge among
tasks with similar difficulty (Parisotto et al.] 2015) Yin and Pan 2017 Gupta et al. 2018a] Liu et al.)
2019 Tao al., ) and (2) learning multiple reusable skills solve complex task divide-
and-conquer manner (Bacon et al. 2017 Frans et al. [011 Nachum et al.) 2018a Eysenbach et al.
2018 Kim al. 202 2
al. 2021 These methods allow an agent to learn a new task with fewer training samples. However, demonstrations are too task-specific to be reused in a new task, and incorporating external guidelines from different sources into existing knowledge-reus frameworks is not straightforward. Moreover, current
from-ddeoosstration and knowledge-reuse approaches lack the flexibility to rearrange and recompose different demonstrations or knowledge, so they cannot dynamically adapt to environmental changes.
In this work, we introduce Knowledge Grounded Reinforcement Learning (KGRL), a novel problem with the following goal: An RL agent can learn its own policy (knowledge) while referring to external knowledge. Meanwhile, all knowledge can be arbitrarily recomposed, rearranged, and reused anytime in the learning and inference stages. A KGRL problem simultaneously considers the following three questions: (1) How can an agent follow a set external knowledge from different sources? (2) How can agent efficiently learn new knowledge by referring to external ones? (3) What is proper

---

Under review as a conference paper at ICLR 2023
representation of knowledge such that an agent dynamically recompose, rearrange, and reuse knowledge in both the learning process and model deployment?
To address the challenges in KGRL, we propose o simple yet effective actor model with embedding- based attention mechanism. This model is orthogonal to training algorithms and easy to implement. In this model, each internal-knowledgee policy is paired with one learnable embedding. An internal- knowledge policy, which the agent learns by itself, is also paired with one learnable embedding. Then another learnable query embedding performs an attention mechanism by attending to each knowledge embedding and then decides which knowledge to follow. With this attention mechanism, addition, the knowledge
the agent can learn new skills by following set of external guidance. and query embeddings design disentangles each knowledge from the selection mechanism. so set of knowledge can dynamically recomposed or rearranged. Finally, all knowledge policies are encoded into joint embedding space, so they can be reused once their embeddings are appropriately learned.
We evaluate our proposed KGRL method for grid navigation and robotic manipulation tasks. The results demonstrate that our method answers all the questions considered in KGRL. In analyses, we show that the proposed approach achieves sample efficiency, generalizability, compositionality, and incrementality, which are the four essential components of efficient learning (Kaelbling, 2020). At the same time, method also enables behavior interpretability because of the embedding baased attention mechanism.
Our contributions
We introduce KGRL, a novel RL problem addressing how an agent can repeatedly refer to arbitrary set of external knowledge for learning new skills.
We propose novel actor model that achieves the goals of KGRL and is orthogonal to training algorithms.
We demonstrate in the experiments that our KGRL method satisfies the four properties of efficient learning (Kaelbling 2020) and learns interpretable strategies.
2 RELATED WORK
Several lines of research in RL have studied how an agent can incorporate external demonstrations/egg. ulations into learning learn reusable policies. Here we summarize the differences between their research goals and the ones in the KGRL formulation.
Learning from demonstrations/exteraa strategies. Providing expert demonstrations or external strategies to an RL agent is popular way to incorporate external guidance into learning. Demonstra- tions are (sub-)opting )optimal examples of completing a task and represented as state-action pairs; external strategies are (sub-) (ubb))optimal policies that are represented in specific form, e.g., fuzzy logic. Previous work has leveraged demonstrations or external strategies by introducing them into the policy-update
steps of RL (Hester et al.) 2017: Rajeswaran et al.) 2017
Vecerik et al.) 2017 7 Nair et al.) 2018
Pfeiffer et al. 2018 Goecks et al. 2019 Kartal et al. 2019
Zhang et al. 2220 2021) or performing
imitation learning (IL) (Abbeel and Ng 2004 Ross et al.
201 Ho and Ermon 2016 Duan et al.)
2017 Peng et al., 2018a b] Ding et al. 2019). Learning from demonstrations (LrD) and II. require sufficient number of optimal demonstrations, which are difficult to collect, to achieve sample-effciint learning. Moreover, those demonstrations are task-dependent and can hardly be reused in other tasks. Compared to LfD and IL, learning from external strategies, including KGRL, can achieve efficient learning by referring to highly imperfect knowledge. Also, KGRL can incorporate general knowledge that not necessary task-related. Finally, our KGRL approach learns uniform representation of mnwleddee in different forms, so it supports flexible hatmmpppiiioeeeaaaaggeeeeeeeeeee arbitrary set of knowledge. These abilities are not all achievable by previous work in this direction.
of an
Safe RL. Other than demonstrations, regulations are also essential guidance that can help an agent achieve effective learning. Safe RL, which incorporates safety regulations (constraints) into learning and inference stages, has gained more attention the past decade. Current research in safe RL mainly includes safety in learning through (1) jointly optimizing the expected return and safe-related
cost (Ammar et al. 2015 Achiam al. 2017 Chow al. 2018 Stooke et al., 2020 Ding et al.
2

---

Under review as a conference paper at ICLR 2023
2021 and (2) restricting exploration in safe space Gaariaa and Fernández 2012 Berkenkamp et al. 2017 Alshickh et al.) 2018: Cheng et al., 2019; Thananjeyan et al. 2021 Liu et al. (022).
The constraints in safe RL can be softly or strictly imposed, where the tormer aims decrease the number of safety violations, and the latter prohibits any of them in both learning and inference stages. Compared to KGRL,, safe RI. usually does not consider reusing the safety constraints in new task or dynamically adjusting regulations according to the environmental changes.
Knowledge transfer and meta-RL. How an agent can transfer previously learned knowledge to
new task is a well-recognized challenge. Several studies have focused on transferring the knowledge
learned in a single task to a multi-task policy (Parisotto et al. 2015 Rusu et al. 2015 Yin and Pan
2017 Xu et al. 2020. Tao et al. 2021). On the other hand, line of research in meta-RI Seeta-RL. investigates
how to quickly adapt the knowledge learned in multiple tasks to a new one (Finn et al. 2017: Gupta
et al. 2018a 8a b) Nagabandi et al. 20188) Liu et al., 2019). These two research directions do not enable incorporating external guidance from different sources into learning, so their transferred knowledge is limited to the one learned among similar environments (Glatt et al. 2016). In addition, they require extensive amount of training data before a set of knowledge can be reused in new task. Compared to previous work in knowledge transfer and meta-RL, KGRL allows an agent to follow and reuse an arbitrary set of knowledge obtained from different sources in a new task, and these sources can be very different from the new task. Moreover, our KGRL approach does not require separate process, such as policy distillation (Rusu et al.] 2013), or numerous training samples collected from multiple environments to learn transferable knowledge.
Hierarchical RL (HRL). Other than transferring knowledge among tasks with similar difficulty,
another popular line of research, HRL, aims to decompose complex task into hierarchy of subtasks and learn a reusable skill for each subtask. Previous research in HRL has focused on learning a
hierarchy of complex task (Dayan and Hinton 1992 Sutton et al. 1999 Kulkarni al. 2016 Levy (Stolle
eet al. 2017 Nachum et al.] 2018b] [Tiagg et al. 2019). discovering options//ubraskss/uubbus
and Precup 2002; Bacon et al. 2017 Vezhnevets et al. 2017 Nachum et al. 201 20] Riemer et al.
2018 Eysenbach et al. 2018 Khetarpal and Precup 2019) Zhang and Whiteson 2 Kim et al.
2021), and transferring learned skills among different tasks (Konidaris and Bartol 2006; Frans et al.
2017 Tessler et al. 2017 Peng et al. 2019 Qureshi et al.) 2019 Tseng et al. 2202I). Compared to the skills in HRL, knowledge in KGRL can be partially contained in, be more complex than, or even has little to do with the task. That is, it is not limited to a skill that solves subtask. This property enables flexibility of providing knowledge and improves exploration in an unseen environment. In addition, skill in HRL is usually assigned with fixed index a task-dependant subgoal, making an agent difficult to dynamically recompose or rearrange skills in learning or inference stages. On the other hand, our KGRL method disentangles knowledge representation from the high-level selection mechanism of a task, making an agent able to dynamically adapt to environmental changes anytime.
Program- Program-guided RL. Similar to HRL, program-guided RL also decomposes a complex task into subtasks and learns subpolicies for them. However, instead of learning a hierarchy of the task, rogram-guided RL guides an agent to complete task with a program. which explicitly specifies the flow of executing the subtasks given the environmental conditions (Sun et al., 2019:; Brooks et al. 2021 Trivedi et al. 2021 Yang al., 2021 Zhao et al. (2221). This guidance requires a structured and unambiguous hierarchy of a task and is more task-dependent compared to knowledge in KGRI.
Moreover, to reuse previously learned subpolicy, program-guided RL needs to first generate new program with a new control flow, hence lacking the flexibility to dynamically recompose and rearrange subpolicies.
3 KGRL PROBLEM FORMULATION
In this section, we formally define the problem formulation of Stowledge-Groundde Reinforcement Learning (KGRL). We consider the family of discrete-time finite-horizon Markov Decision Processes (MDPs) with knowledge guidance for an environment. Such an MDP is formulated as a tuple (S,A, G, 7))), where S is the state space, A is the action space, T : S A x S + R is the transition probability distribution, & is the knowledge set, R is the reward function, p is the initial state distribution, and y is the discount factor.
3

---

outputs an Under review as a conference paper at ICLR 2023
action (at)
Actor
Environment
π im
Kin
Km
T Ø1
K Knowledge (92)
k Knowledge (93)
Inner Actor Qin
Knowledge (91)
Ok
state (st)
Figure 1: The proposed model architecture of a KGRL agent.
Each element in the knowledge set, 9 E G, can be general knowledge, such as commonsense, suggestion, guideline, principle, specification, rule, restriction, described a knowledge mapping Tg S + A. This mapping represents the low-level execution strategy of each knowledge and can be any form, such as hand-crafted rule, fuzzy logic, or a neural network, as long action given state. In our experiments, we focus on g being suggestions in the form of hand-crafted rules or neural networks. This is because flexible recomposttio//earangeeee is fundamental to efficient learning but is currently not achievable by existing methods. Moving forward, this setup can lay a foundation incorporate a more extensive knowledge set.
of set of suggestions
The goal of KGRL agent is to maximize the expected rewards and develop the following attributes: (1) The agent learns its own knowledge by referring to external one. (2) The agent can recompose/re- arrange/reuse knowledge in training and inference stages as well as new environments. (3) The agent's 's behaviors are interpretable in terms of which knowledge refers to at each time step.
4 PROPOSED APPROACH
We propose a novel actor model for KGRL with the concept inspired by the self-attention mecha- nism (Vaswani et al. 2017]. The actor model includes an inner actor and set external knowledge mappings, { Tg. Tg2..... oo. J. The actor architecture is illustrated in Figure [ The idea behind this design is that the actor will decide to follow an external guidance or its own policy given a state E S at the t-th time step. This dynamic decision making should eventually lead the agent to receive higher accumulated rewards based on reward function R. Moreover, for external knowledge and deployed without further training, each knowledge
set to be arbitrarily recomposed/rearrange should be represented in a form that is order-invariant, and the knowledge-selection mechanism should be independent from which set of knowledge is used during training. In the following sections, we introduce the detailed design of each component in the proposed actor model.
4.1 ACTOR MODEL ARCHITECTURE
External Knowledge. An important property a KGRL policy is the flexibility of knowledge coommposiioo/caarmngeenntttttsss
during training and testing phases. To obtain this flexibility, knowledge
we propose a uniform representation for each knowledge so that the components set is order-invariant. We assign learnable key (embedding) for each external knowledge mapping, which is a dy dy-dimensional vector denoted as kg, E IR,,, for j = 1,2,.... n. Note that the knowledge key ky, represents the entire knowledge mapping Ty instead of each state-action pair suggested by this mapping, so kg, does not depend on states or actions. Since each knowledge is encoded as an independent key, which lies in joint embedding space, by adding/removing keys to/from the space to adapt to the dynamical changes in an environment. Meanwhile, one can also combine knowledge keys from different tasks in new learningeevaluation process.
can change the number of knowledge
Inner Actor
An external knowledge set may not contain policies that can directly solve a specific task, agent needs be capable of developing its strategies. Therefore, we add inner actor
4

---

Under review as a conference paper at ICLR 2023
in the actor model of KGRL. An inner actor contains two learnable function approximators oi and Øk. Given a state St, the function approximator Oin predicts an inner policy, Tin = Φin (s:) : + A, and Φk generates an inner key, ki.. c (s;) R.,,, The inner policy serves the same purpose policy in regular RI,, which represents the strategy that an agent learns by itself through interactions with the environment. Similar to external knowledge mapping *g.- the inner policy Tin can be viewed as the internal knowledge mapping with paired learnable inner key kin. Note that unlike an external knowledge key, the inner key varies according given state, because the inner strategy will change as the agent learns from more experiences over time.
Query. Given the external and internal knowledge mappings, an agent needs to decide which knowl- edge to refer to. Importantly, this selection mechanism should be disentangled from knowledge set to support its flexible eecomposition/eeaaraggnno
ability. That is, an agent should still be able to select proper knowledge mapping when different knowledge set is provided. To achieve this goal, we use an attention mechanism to query knowledge. This attention mechanism is implemented by adding another learnable function approximator Pu that generates query vector, denoted as u= du (Sx) (Sx)) E Rd given a state Sy. The query will be attended to the keys to calculate the weights for each knowledge mapping. Note that the proposed model predicts the query and keys separately instead of directly predicting the weight for each guidance. In this way, we disentangle the selection component (the query) from the knowledge set (the keys). The following section describes how we compute the weights.
4.2 ACTION PREDICTION
Attentions. We compute the attention of the query u over the inner keys and external knowledge keys by their dot products. The results become the weights for the inner and external guidance:
Win 11 Kin
Wg; = kgj
Win is larger, then the agent more likely to adopt the inner policy. On the other hand, if way is larger, then the agent is more likely to follow the guidance from the j-th external knowledge.
Final Policy. A straightforward way to describe the final policy given the weights W/n and Wg and their corresponding knowledge mappings Tin and Tg for 1.2...... n. is
a. T T--(s)),
x(1 = win Tin (in(ss) +
Wgg u (-1st),
which is a weighted sum of each strategy. We apply equation (3) to environments with a discrete action space, where each knowledge mapping predicts probability vector. However, found that in environments with a continuous action space, taking actions sampled from equation will guide an agent complete task less efficiently.
We explain by simple example why computing weighted sum is less favorable choice when integrating multiple knowledge map- pings in a continuous action space. Figure]ssowss a 2D environment where an agent navigates avoid the obstacle and reach the goal. The agent is provided with two knowledge mappings. Their sug- gested actions the current time step are plotted as an arrow orange and blue respectively, and their weights are both 0.5. The weighted sum of these two action candidates plotted as a green arrow, which guides the agent to go directly into the obstacle. This example demonstrates that weighted sum of multiple actions in a continuous action space can lose critical information contained in individual actions and lead to worse behaviors. Hence, we instead sample only one action candidate according to the weights.
KB1+KG2
KG2
KG1
Agem
Figure 2: An example incor- porating multiple actions by weighted sum. It shows that weighted sum of actions can lead the agent to go into the obstacle (the gray wall).
To sample an action suggested by one of the knowledge mappings while making the entire actor model differentiable, we first sample an
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Knowledge-Guided Reinforcement Learning (KGRL) with Our Actor Model
Input: environment & with MDP (S, A. T, R, p. Input: external knowledge mappings gg T2 ".y Fg
initialize kg, for j = 1,,,,.... .2,,...
initialize Pin: 01: Du
for each time step do
Generate kg....., k kini Tins
Compute Win, Wg. ........ Wg according to equation 1
Sample action a, T(-[s;) according to equations 2
Apply a, and observe a new state and reward 1,41,7 from 3
Update Din; tk, Pus kg ......., kg. by any RL algorithm
return Pins Øk Pur K kg...... Kg.
element from the set (in, 91,92; ..... 9n according the weights [Winy Wo:: 10 g2 W using Gumbel softmax (Jang et al. 2016):
~ saaibeelssftaaaeee (Win Wgg: (9g2 Wgn.)
Then given a state St, an action is sampled from the knowledge mapping म. (-Iss) using the reparame- terization trick, where de can be one of {din, .......,, ,g. according to the sampled e.
4.3 LEARNING ALGORITHMS
The proposed actor model can be end-to-end learned with any existing RL algorithm, and the implementation simple. Algorithm D describes our proposed KGRL algorithm.
EXPERIMENTS
We validate the proposed KGRL actor model on two sets of widely used environments with different
properties: MiniGrid Cheeallir--Boiveee et al.) 2018) and OpenAI-Robotie (Plappert et al. 2018).
Through experiments on these environments, we answer the following four questions: (1) Can our KGRL agent learn effective new knowledge by referring
arbitrary set of external guidance? (2) Can the knowledge policies and their learned embeddings be recomposed, rearranged, and reused to solve new task? (3) our KGRL method sample efficient and generalizable? (4) Does our KGRL agent learn interpretable behaviors? The implementation details can be found in Appendix.
Baselines. We compare our KGRL method with the following basclines that incorporate external guidance in different ways. (1) Behavior cloning (BC): An agent follows only the external knowledge to solve a task. The results of this strategy can be seen as the upper bound of BC, where the demonstration data is generated by applying the external knowledge. (2) RL: An agent learns policy from scratch by RL. No external guidance is incorporated. (3) RL+BC: An agent learns a policy by RL with BC signals provided from external knowledge. We follow the method proposed in Nair et al. (2018) to incorporate BC losses into RL.
5.1 MINIGRID
We evaluate all methods the following MiniGrid environments with discrete action space and sparse rewards: Empty-5x5, Hmmpt--Random-5x5 Empty-16x16, Unlock, DoorKey- 5x5, and DoorKey-8x8, Among them, DoorKey are the environments that an agent needs to pick up key, open a door, and then reach goal, which can use the skills learned in Unlock (pick up a key and open door) and Empty (reach goal). We choose these environments to study if our method can recompose/reuse previously learned knowledge. The observation space of these environments the directed first person view with a range of 5x5 grid, and the action space has seven discrete actions.
Environments.
Setup. We compare BC and Proximal Policy Optimization algorithm (Schulman et al. 2017 7) under the
RI. (PPO), RL+BC (PPO-BC), and KGRL (PPO-KGRI) formulations. In PPO-KGRI....

---

Under review as a conference paper at ICLR 2023
PPC k
1P0
1910
PP 40
-P3-CC
DC
PPO DC
1PO-CC
IPO RORL000351
191 6ORL2005511
14. KORLOC0S1
50
12
Tirectope
Timeetoge
Tireddepe
(a) Empty-5x5
(b) Sainpty-aando--S5 5x55
(c) 1mpty-16x16
PPO
PR0 GGALMG
PRO
PPO MORUUGGLG 40G3]
PPO
PPO DOOLLGR
1PO KGRiia
00
50
1900
12301 HERO 20000 20000 1000
20008
40:00
6002
1009
0.
1.30
(d) Unlock
(c) DeorKey-5x5
( DoorKey-8x8
Figure 3: The learning curves of different agents in MiniGrid environments. Our KGRL method achieves better sample efficiency in most of the environments. Also, adding redundant knowledge in the proposed model does not affect the performance.
inner actor model same as the actor model of PPO and add the learnable inner key, knowledge keys, and query. We define three simple knowledge policies for the MiniGrid environments. (KG1) pick up a key: if there is key in the observation, "go in the direction" of the key; if the key is just in front of the agent, pick up the key. (KG2) open door: if there is door in the observation, "go in the direction" of the door; if the door is just in front of the agent, open the door. (KG3) reach goal: there is goal in the observation, "go in the direction" of the goal.
Results. Figure 3 shows the training results of the baseline methods and PPO-KGRL with different knowledge combinations. In most of the environments, PPO-KGRL achieves significantly better sample efficiency given sub-optimal knowledge. Also in general, the improvement is larger when the task is more complex (Empty < Unlock DoorKey). Note that PPO does not show any success in DoorKey-8x8 within three million steps. However, we found that in Empty-16x16 16, our KGRL agent learns more quickly at the beginning but slows down its progress over time. The reason might be that 16x requires only exploration instead of other skills, the agent cannot benefit too much by following external guidance. Moreover, in a small environment such as DoorKey-5x55 the advantages of providing external knowledge is difficult demonstrate due to extra learnable parameters in our actor model.
We also study the effects of different knowl- edge combinations for PPO-KGRL. In Un- lock, we found that KG2 is more helpful than KG1; in DoorKey-8x8, KG3 is more help- ful than KG1+KG2. We conjecture that the knowledge which directly helps receive the rewards guides an agent more effectively in these environments. Also, as shown in Fig- a-dd) adding redundant knowledge will not affect the performance. This observation demonstrates the benefit of disentangling the knowledge embeddings and query, thus in- creasing the flexibility of providing external guidance.
__pick_ipp key
wwrcach. .00l
Stap E: pick ba
Step I5:
Step 22: exarch for geal
Stpp 29: meach gasl
Stape
Figure 4: The interpretation of the agent's 's behaviors. The left image shows the (unnormalized) weights for each knowledge.
7

---

Under review as a conference paper at ICLR 2023
Policy Interpretation. Figure 4 shows that our KGRL method learns interpretable agent behaviors. The weight changes indicate that in a DoorKey-8xe environment, the agent chooses to sequentially "pick up the key", "open the door", "search for the goal", and "reach the goal". Note that the attention to the inner policy does not disappear throughout the entire episode. This shows that the inner policy helps agent explore the environment.
Analysis of Generalizability, To test the generalizability of PPO and PPO-KGRL, we train an agent an empty environment with fixed goal (Empty) and evaluate the agent an empty environment with a randomized goal (Empty-Randone memorizes the goal position or really understands how to find the goal. As shown in the left column of Table T the returns of PPO- KGRL achieve an average of 0.94 and a minimum of 0.77. This is much closer to the upper bound, which is the average return of a PPO-KGRL agent trained on the Empty-Randome environment.
This experimental design verifies whether the agent
Init. Env.
Empty
DoorKey-5x5
DoorKey- 8x8
Test Env.
Simpty-Randon
Empty
Empty- Raanomm
Unlock
Unlock
PPO
0.92 (0.71)
0.55 (0.05)
0.79 (0.03)
0.43 (0)
0 (0)
PPO-KGRL
0.94 (0.77)
0.88 (0.55)
0.90 (0.55)
0.54 (0)
0.85 (0)
Upper Bound
0.95 (0.84)
0.93 (0.87)
0.95 (0.84)
0.94 (0.84)
0.94 (0.84)
Table I: Zero-shot evaluations on three environments (Empty, Empty-Random, Unlock) of RL agents trained on an initial environment (Empty and DoorKey) using PPO or PPO-KGRL algorithms. The average and minimal returns are shown for each evaluation. Our KGRL method is more generalizable and effective in transferring previously learned knowledge.
Analysis of Reusing, Recomposing, and Rearranging Knowl- edge. For our KGRL method, we verify if the external and internal knowledge policies and their embeddings can be reused by the following two sets of experiments: (1) zero-shot transfer- ring the knowledge policies and embeddings to a simpler task and (2) initializing the external guidance by those knowledge policies and embeddings when learning a more complex task. The second set of experiments also verifies if the knowledge policies and embeddings can be recomposed and rearranged.
PPG
PTO ROD--GGRUUnnsckK
PRO-KGRL(M))
FPO -GR (tw.ptt
3.4
Timesdepp
In the first set of experiments, we zero-shot evaluate an agent trained in DoorKey environments on Empty, Eimpty-Randon and Unlock environments. The four columns on the right side of Table [ show the average and minimal returns. We found that an agent trained with PPO-KGRL significcal outperforms the one trained with PPO in all environments. These results demonstrate that our KGRL policies have better reusability.
Figure 5: The knowledge reusabil- ity experiments on MiniGrid- DoorKey-8x8, Our KGRI. method further improve sample efficiency by reusing previously learned knowledge.
In the second set of experiments, train three PPO-KGRL agents with external knowledge being:
1. all: KGI + KG2 + KG3. This set of knowledge is the same one used in Figure 3
2. Unlock (reused) + KG3: the inner knowledge policy and embeddings learned in Unlock + KG3, This set of knowledge recomposes and rearranges the first set of knowledge.
3. Unlock (reused) + Empty (reused): the inner knowledge policy and embeddings learned in Unlock + the ones learned in Empty-Randon This set of knowledge recomposes and rearranges the second set of knowledge.
Figure 5 shows the training results of PPO and PPO-KGRL with different sets of external knowledge. The two KGRL agents that reuse previously learned knowledge perform better than PPO and PPO- KGRL(all). Also surprisingly, we found that the number of samples required by PPO-KGRL that reuses the Unlock knowledge plus the one required by learning the Unlock knowledge is even less than the required by (al)), which learns with KGI+KG2+KG3, These results indicate
8

---

Under review as a conference paper at ICLR 2023
Success Rate
Rale
SAC
Success
BC SAC
SM
DC
54C K2
SAC SAC KGRL
SAC KOTAAal
Timesteps
Timesteps
Range Goal (in ratio)
Range of Goal (in ratio)
(a) Reach.
(b) Pick-And-Place
(a) Reach.
(b) Pick-And-
Figure The learning curves of different agents in Ope- nAI robotic environments. Our KGRL method leads to better sample efficiency and stability in both environ- ments.
Figure Results the policy trained on 0.1x the original goal range. Our KGRL method is less affected by the changes the environment.
that with the abilities to flexibly reuse, recompose, and rearrange knowledge, our KGRL method can further enhance the training efficiency.
5.2 ROBOTICS
Environments. We also verify the proposed KGRL approach in the following OpenAI robotic enviromments with continuous action space and sparse rewards: Reach and Pick-And-Place Both environments contain task of reaching a randomly sampled goal. Similar to the MiniGrid envi- ronments, these two tasks can demonstrate the ability of our KGRL method reusing, recomposing, and rearranging previously learned knowledge when solving a new task. An observation these environments includes the position and velocity of the end-effector and object (if it exists) and the position of the goal. An action of these environments contains the position variation of the end-effector and the distance between the two fingers.
Setup. We compare BC and the soft actor-critic (Haarnoja et al. 2018) algorithm with hindsight experience replay (Andrychowicz et al. 2017) under RL (SAC), RL+BC (SAC-BC), and KGRL (SAC-KGRI)) formulations. In BAC-KGRL. query on top of the actor model of SAC. We define the following two external knowledge policies for the OpenAl robotic environments, where Pee E R the position of the end-efffcctor and Potj g is the position of the object. (KG1) move straightly to the goal: if ||Pee Pobj l| < E, move straightly the goal with the gripper closed; otherwise, stay unmoved. Note that there is no object in Reach, so is set to infinity in this environment, and KG1 always guides the agent towards the goal. (KG2, for Pick-Ande Bick-And-Place only) move straightly to the object: Pee Pabyll 2 E, move straightly to the object with the gripper opened; otherwise, stay unmoved.
add the learnable inner key, knowledge keys, and
Results. 6 the training results of Reach and Pick-And-Plas In both environments, SAC-KGRL requires less samples succeed and shows more stability. Compared to SAC-BC. our method better utilizes external guidance regardless of its quality. Moreover, the success of SAC-KGRL in Pick-And-Place demonstrates that our KGRL agent can learn an effective strategy given sub-optimal knowledge. We also study the effects of different knowledge combinations in Pick-And-Place. The learning curves indicate that using both KGI and KG2 leads to slightly better sample efficiency than using only one of them, but there is significant difference among the three combinations.
Analysis of Generalizability We analyze if our KGRL policies are more generalizable by training in both environments with a smaller goal range and testing with larger goal ranges. The training goal range is 0.1 times smaller than the original one, and the testing goal ranges are the original one multiplied by 0.1 to 1. Figure 7 shows the generalizability results. The success rate of SAC and SAC-KGRL agents both decreases as the testing goal range increases, but the SAC-KGRL agent in Reach and the SAC-KGRL(al)) agent in Pick-And- Place less affected by the changes, especially when the goal ranges are larger. These results suggest that an KGRL agent has more potential to generalize to unseen environments.

---

Under review as a conference paper at ICLR 2023
Environment
Reach
Fick-And-Pan
Knowledge
All
IA
KGL
All
IA KGI
IA KG2
IA
KGI KG2
Success Rate
0.99
0.99
0.99
0.99
0.05
Table 2: Evaluation results of removing part of the knowledge set of our KGRL policies (IA: inner actor, All: IA + all KGs). Our proposed method enables the inner actor mimic the strategies of external knowledge, and may become a more powerful piece of new knowledge.
Analysis of Knowledge Control Flexibility. We demonstrate the flexibility of our method re- composing and rearranging knowledge by removing part of the knowledge set during evaluation. We evaluate Reach with the policies learned by SAC-KGRL and Pick-And-Place with the policies learned by SAC-KGRL(al)). Table (oole2]hook shows the testing results of both tasks after removing part of the knowledge set. The learned inner actors in both environments achieve high success rate without the external guidance, indicating that KGRL agent learns not only follow the guidance but also mimic its strategy. This imitation process allows an agent to develop an inner policy that outperforms external strategies, and itself can then become new knowledge useful for solving other tasks.
Model
SAC
SAC KGRL-all
SAC-KGRL-KGI
100k Training Steps
0.24
0,56
0.25
0.48
200k Training Steps
0.61
0.99
0.99
0.98
Best
0.8
1
1
Table 3:
Zero-shot transfer results of Pick-And-Plant policies to Reach. Our proposed method is more effective and efficient in reusing learned knowledge.
Analysis of Knowledge Reusability. To demonstrate that our KGRL method enables better knowl- edge reusability, we apply zero-shot transfer from the learned Pick-And-Place policies to Reach. Note that the observation spaces of Reach and Pick-And-Place are different since there is no object in Reach. Hence, we apply zeros as the object's 's relative position and velocity Reach to unify their observation spaces. Table 3 shows the success rate of different Pick-And-Place policies applied to Reach. The results indicate that SAC-KGRL is more efficient in reusing knowledge since the transfers mostly successful after only 200k training steps.
CONCLUSION AND DISCUSSION
In this work, we introduce KGRL, novel RL problem discussing how an agent can follow an arbitrary set of external guidance and learn its own policy. Also, the agent should be able to repeatedly reuse previously learned knowledge and dynamically recompose and rearrange knowledge to adapt to environmental changes. To address these challenges, we propose simple yet effective actor model that performs an embedding- based attention mechanism. The experimental results show that the proposed model achieves the goal of KGRL. Our KGRL algorithm also satisfies the four properties of efficient learning (Kaelbling) 2020) and learns interpretable agent behaviors.
This work serves as initial effort for the ultimate goal of KGRL, which is to automatically collect a large group of knowledge from different sources, such as books, the Internet, or previous training processes. This group of knowledge can be shared among multiple environments/tasks and continuously expanded, and a wide variety of artificial agents can use it as their guidance. Moving forward, the following research directions KGRL are worth exploring: (1) incorporating restrictions that should be strictly followed during the learning and inference stages, (2) composing knowledge with different state and action spaces, and (3) dealing with more complex relationships among different knowledge, such as conditional dependence and conflicts. We hope this work can motivate future research KGRL.
REFERENCES
Pieter Abheel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-frrt international conference on Machine learning, page 1, 2004.
10

---

Under review as a conference paper at ICLR 2023
Joshua Achiam, David Heldd Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 11. PMLR, 2017.
Mohammed Alshickh, Roderick Bloem, Rüdiger Fhlers, Bettina Känighofer, Scott Nickum, and Ufuk Topeu. Safe reinforcement learning via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Haitham Bou Ammar, Rasul Tutunov, and Eric Eaton. Safe policy search for lifelong reinforcement learning with sublinear regret. In International Conference on Machine Learning, pages 2361-2369, PML.R. 2015.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model- based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30, 2017.
Aude G Billard, Sylvain Calinon, and Rüdiger Dillmann. Learning from humans. Springer handbook of robotics, pages 1995-2014, 2016.
Ethan Brooks, Janarthanan Rajendran, Richard Lewis, and Satinder Singh. Reinforcement learning of implicit and explicit control flow instructions. In International Conference on Machine Learning, pages 1082-1001 1091. PMLR, 2021.
Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3387-3395, 2019.
Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai com
gym. httpp::/ http://ww...oommm.a..a.iaaeeeeee http://wxxiimcc//mmmmmmm http://gggmmmmmmmgee mini-rinniiiii 2018.
Yinlam Chow, Ofir Nachum, Edgar
and Mohammad Ghavamzadeh. A lyapunov-base approach to safe reinforcement learning. Advances in neural information processing systems, 31, 2018.
Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. Advances in neural information processing systems, 5, 1992.
Dongsheng Ding, Xiaohan Wei, Zhuoran Yang. Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. International Conference on Artificial Intelligence and Statistics, pages 3304-3312. PMLR, 2021.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019.
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017.
Benjamin Bysenbach, Abhishck Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802 1802.06070, 2018
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In JCML, 2017.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. arXiv preprint arXiv: 2017.
Javier Garcia and Fernando Fernández. Safe exploration of state and action spaces in reinforcement learning. Journal of Artificial Intelligence Research, 45:515-564 564, 2012.
Ruben Glatt, Felipe Leno Da Silva, and Anna Helena Reali Costa. Towards knowledge transfer in deep reinforcement learning. In 2016 5th Bruzilian Conference Intelligent Systems (BRACIS), pages 91-96. IEEE, 2016.
Vinicius G Goecks, Gregory M Gremillion, Vernon J Lawhern, John Valasek, and Nicholas R Waytowich. Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments. arXiv preprint arXiv: 1900.0428,, 2019.

---

Under review as a conference paper at ICLR 2023
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-learning for reinforcement learning. arXiv preprint arXiv:/806.00 04640, 2018a.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Picter Abbeel, and Sergey Levine. Meta-reinforcement
learning of structured exploration strategies. Advances in neural information processing systems, 31, 201
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Arold, Ian Osband, John Agapiou, al. Learning from demonstrations for real world reinforcement learning. 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax arXiv preprint arXiv: 1rXiv:161 1,01 140, 2016.
Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Leslie Pack Kaelbling. The foundation of efficient robot learning. Science, 369(6506): 916, 2020.
Bilal Kartal, Pablo Hemandezz-Laal, and Matthew E Taylor. Action guidance with mets for deep reinforce- ment learning. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 15, pages 153-159, 2019.
Khimya Khetarpal and Doina Precup. Learning options with interest functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9955 9956, 2019.
Jackyeom Kim, Seohong Park, and Gunhec Kim. Unsupervised skill discovery with bottleneck option learning. arXiv preprint arXiv:2106 14305, 2021.
George Konidaris and Andrew Barto. Autonomous shaping: Knowledge transfer in reinforcement learning. In Proceedings of the 23rd international conference Machine learning, pages 489-496, 2006.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Sacedi, and Josh Tenenbaum. Hicrarchical deep reinforcement leaming: Integrating temporal abstraction and intrinsic motivation. Advances neural information processing systems, 29, 2016.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Leaming multi-level hierarchies with hindsight. arXiv preprint arXiv:1712.0m 104,00088 2017.
Hao Liu, Richard Socher, and Caiming Xiong. Taming maml: Efficient unbiased meta-reiinforeemet learning. In International conference machine learning, pages 4061-4071. PMLR, 2019.
Puze Liu, Davide Tateo, Haitham Bou Ammar, and Jan Peters. Robot reinforcement learning the constraint manifold. In Conference Robot Learning, pages 1357-1366. PMLR, 2022.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv: 2018a.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018b.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through metf-reiinorrement learning. arXiv preprint arXiv:1803.1 11347, 2018
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Picter Abbecl. Overcoming exploration in reinforeement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6292-6299. IEEE, 2018.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimie: Deep multitask and transfer reinforce- ment learning. arXiv preprint arXiv: 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Iuca Antiga, et al. Pytorch: An imperative style, high-performance deep learning lihrary. Advances in neural information processing systems, 32, 2019.
12

---

Under review as a conference paper at ICLR 2023
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4)11-4, 2018a.
Xue Bin Peng. Angjoo Kanazawa, Sam Toyer, Picter Abbeel, and Sergey Levine. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow, arXiv preprint arXiv: 1810.0021) 2018b.
Xue Bin Peng. Michael Chang. Grace Zhang, Pieter Abbeel, and Sergey Levine. Mep: Learning composable hierarchical control with multiplicative compositional policies. Advances Neural Information Processing Systems, 32, 2019.
Mark Pfeiffer, Samarth Shukla, Matteo Turchetta, Cesar Cadena, Andreas Krause, Roland Siegwart, and Juan Nieto, Reinforced imitation: Sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations. IEEE Robotics and Automation Letters, 3(4):4423 4430, 2018.
Matthias Plappert, Marcin Andrychowicz, Alex Ray. Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint (eX.iv::802.000000 2018.
Ahmed Qureshi, Jacob J Johnson, Yuzhe Qin, Taylor Henderson, Byron Boots, and Michael C Yip. Composing task -gnoostc policies with deep reinforcement learning. arXiv preprint arXiv: 1905. 10681, 2019.
Antonin Raflin, Ashley Hill. Adam Gleave, Anssi Kanervisto, Maximilian Emestus, and Noah Dormann. Stable- baselines3: Reliable reinforcement learning implementatiions. Journal of Machine Learning Research, 22 (268):1-8, 2021. URL
1364. html
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. arXiv preprint UrXiv:1709.100
2017.
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. Advances in neural information processing systems, 31, 2018.
Stéphanc Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings. 2011.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv:151..0625.. 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Martin Stolle and Doina Precup. Learning options in reinforeement learning. In International Symposium on abstraction, reformulation, and approximation, pages 212 223. Springer, 2002.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133-9143, PMLR, 2020.
Shao-Hua Sun, Te-Lin Wa, and Joseph Lim. Program guided agent. International Conference on Learning Representations, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: introduction. MIT press, 2018.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforeement learnin.. Artificial intelligence, 1 12((-2):181-21 1999.
Yunzhe Tao, Sahika Gene, Jonathan Chung, Tao Sun, and Sunil Mallya. Repaint: Knowledge transfer in deep reinforcement learning. In International Conference Machine Learning, pages 10141-10152. PMLR, 2021.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hierarchical approach to lifelong learning in minecraft. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang. Joseph F. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEFE Robotics and Automation Letters, 6-54449,,, 2021.
13

---

Under review as a conference paper at ICLR 2023
Dweep Trivedi, Jesse Zhang. Shao-Ilua Sun, and Joseph Lim. Learning synthesize programs interpretable and generalizable policies, Advances Neural Information Processing Systems, 34:25 14665513,, 2021.
Wei-Cheng Tseng, Jin-Siang Lin, Yao-Min Feng, and Min Sun. Toward robust long range policy transfer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9958-9966 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. In NIPS, 2017.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards, arXiv preprint arXiv:/707 088 2017.
Alexander Sasha Vezhnevets, Simon Osindero, Schaul, Nicolas Heess, Max Jaderberg. David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pages 3540-3549 PMLR, 2017.
Zhiyuan Xu, Kun Wu, Zhengping Che, Jian Tang, and Jieping Ye. Knowledge transfer in multi-task deep reinforcement learning for continuous control. Advances in Neural Information Processing Systems, 33: 15146-15155 15155, 2020.
Yichen Yang, Jeevana Priya Inala. Osbert Bastani, Yewen Pu, Armando Solar-Lezama, and Martin Rinard. Program synthesis guided reinforeement learning for partially observed environments. Advances in Neural Information Processing Systems, 34, 2021.
Haiyan Yin and Sinno Jialin Pan. Knowledge transfer for deep reinforcement learning with hierarchical experience replay. In Thirty-First AAAI conference on artificial intelligence, 2017.
Peng Zhang, Jianye Hao, Weixun Wang, Hongyao Tang, Yi Ma, Yihai Duan, and Yan Zheng. Kogun: accelerating deep reinforcement learning via integrating human suboptimal knowledge. arXiv preprint arXiv:2002.07416 2020.
Ruohan Zhang, Faraz Torabi, Lin Guan, Dana Ballard, and Peter Stone. Leveraging human guidance for deep reinforeement learning tasks. arXiv preprint arXiv: 9999099006 2019.
Shangtong Zhang and Shimon Whiteson. Dac: The double actor-critic architecture learning options. Advances in Neural Information Processing Systems, 32, 2019.
Yichuan Zhang, Junkai Ren, Junxiang Li, Qiang Fang, and Xin Xu. Deep leaarnngg with explainable and transferable domain rules. In International Conference on Intelligent Computing, pages 259 273. Springer, 2021.
Zelin Zhao, Karan Samel, Binghong Chen, et al. Proto: Program-guidee guided transformer for rangram-guided tasks. Advances in Neural Information Processing Systems, 34, 2021.
14

---

Under review as a conference paper at ICLR 2023
A
EXPERIMENTAL DETAILS
A.1
MINIGRID ENVIRONMENTS
A.1.1
ENVIRONMENTAL DETAILS
The environmental details can be found at httppp////..aaa..c.ccaaaaaaaaaaa mtg.
A.1.2 PREDEFINED KNOWLEDGE
KGI, KG2, and KG3 for MiniGrid enviromments hand-crafted rules that output probability vector given state, provide our code defining the rules here.
agent_ pos (abs. thape. (1]7/2 7/2, obs. dos.inago.atann (2]-1)
expest_actions 2eroa (obs.inagee www.ga.................... shape lan poxpert_ruiesi Zen factiona)), devicas' cuda:0r)
def convert ptwpert_pss_t__ddeo s____dd___iiioaa___eeeee_ aotis
start, coals, sds_co_renoveeoon
img_id_ncet__oe cec.___ct___oodtiioo o 0] [istart gosists,1]. nccnzcro()]
ixpert_acticns iig__id_meett___.meooeoeop mww.eemmmiiionnuue....... www.elddllciiioooooo ""ttinns""igght""
imo_ii__css_connttiio coals];, [ssaast sosia[t,1]..im
10
ine iin_ia_neet_condiiio condittion goalsl:, [1, [Istart
11
expert_acticas [ing______________i_______ [i___i__me____________ ____et__cmdiiiinn.. wtt.:iionnule_ddeeeooooo wtt__iidacciioss ("ddaaccctos""""aaaee"""""""
ide_to_rcmove 15 aot None:
13
sssert_actionn cocccioonnadaaa..aaoooo.. [ida_to_d re_oooe sdle_id, ["Righee" 0
14
expert_actiona tidd____rcoree_________________e renove, Idle_id, actions ["Lfft"]]
15
export_actions (idiionnnsdddd_dd
actions ["Fowwadd]]
17
KG1: pick up a key, 1d:5
18
ir *gul the key" In expeci_
19
sule_ld expert_roles. indes..ndoxe(.M
key")
key_pos ting 5) (o moel
21
key_ 01 poay_po --
22
pos [0]) -ine poyyy a- pos [1]]:_ (ia .nnnaaro(1]
25
wtt.://c..i.itooooooeoiiaee convert tos_to_ ianiic in, poen bey_poa, -Bk_Loa_remeveiing condi,
expert_acticas
25
26
KG2:
door, Ide4
27
"open doee* expert_rsles:
sula_id sxport_rules.inder ("oonn dooz?)
29
pos = (ing =- 4) -Janneero)))
30
doos_peatt, (doos poalt,
agent_
(docr_pos
acent_psa(1)-11 0e
experi weliio/iinnn..... lingiid__dnnd cond ion, rul=_ aut lond [""ggl="]]
33
convert pos__o_________________ so_doo_____ett dit_ddde__itoooeee lons sid agent_pos, door_ dda__oo_snoveeee_eeeeeeee_ee id cond_oondiiion
34
KG3:
goel, id-3
*ga to the ocain
expert_rcles:
37
sule_i id = rulerr..uulesseeee (fgg to the goal";
38
pow (ing e* 8) ()
ponvert_pos_teet do_d__ to_ddrr__ddtoooeee actions tovionnttuie_d
A.1.3 MODEL ARCHITECTURE
We use the same model architecture for all the six environments. Each external knowledge embedding is generated the PyTorchB Paszke al. 2019) module, nn. Embedding, with de 8. The inner actor network. Φin, is 3-layer convolutional neural network followed by I-layer multi-layer perceptron (MLP), Its detailed architecture can found hhttps::s..............tt.. / /ttptthg.tb..t//////// com hwwp//swwii www.wwwem//eemmmmmmmmmmmo / - starter- files The inner key and query networks, &u and Φu. share the same MLP layer with P.n. and cach of them is followed by Tanh activation and output layer with the dimension being dy.
A.1.4 BASELINE ALGORITHMS
BC. Since KGI, KG2, and KG3 in conflict with another, combine these three knowledge mappings (KGI+KG2-KG3) The results of BC obtained by applying KGI+KG2+KG3 in all environments.
PPO-BC. The external knowledge policy KGI+KG2+KG3 can generate demonstrations to guide an agent. With this policy, we add the following BC loss to the objective of PPO:
LBC = cBc CwL (
(5)
where <pc E IR is the coefficient of this loss Dzz (-I1-) calculates the KI. divergence, T(-) is the policy learned by agent, and (-)(-) (-) is the external knowledge policy. We set esc 10 for all environments.
15

---

Under review as a conference paper at ICLR 2023
A.1.5 HYPERPARAMETERS
Each training trial in Section 5.1 is run with 5 different random seeds, and report the average performance of the 5 trained agents for analyses. For the underlying PPO algorithm, follow the implementation of com/ ieswi -
with unchanged hyperparameters.
A.2 OPENAI ROBOTIC ENVIRONMENTS
A.2.1 ENVIRONMENTAL DETAILS
state 8i in unmodified Reach, where 8, E R contains the position and velocity of the end-effector, the distance between the two grippers, and their velocity. A state in Bock-And-Placee where E ut contains the same information that in Reach, the position, rotation, and velocity of the object, and the relative position hetween the object and the end-effector. In modified Reach, which is used in the analysis of knowledge reusability, state Sı, where 8, R225, contains the same information as that Pick-And- Place, but the object's information, except its relative position, is set to O.
An action a in all environments, where aı I contains the position variation of the end-effector and the distance between the two grippers.
A.2.2 PREDEFINED KNOWLEDGE
KGI KG2 for OpenAI robotic environments are hand-crafted rules output mean and standard deviation of an action given a state, We set E to be 0.03 for Pick-And And-Place and the log standard deviation to be -1 for both environments. We provide our code of defining the rules here. Note that we will apply the softmax function to output to make it probability vector.
erv. type
reach!s
kg num
zhrreros Aleaah, KGi, nove atraightly up the guel
kg., antiion_dim))
kg kg_mean__eet mean_actionsiiy O, :31 destred_ goal porrert_ggiip.ps
env_ "fick_and_plsee" kg_nun
10
11
kg_mesn_actions th.zeros (patch_sizo, kg_run, action_sin]]
12
13
pel, gustate obs oobservaa ion'l 6:91, obs n'beeratt ion' I Is, conaqueeze
14
sei_pps_nomm tiniin wwr.iiigoomrr........m dim--1, keepdin-Trusl #batch cize x
15
-Pisx-ind-Placs, XG2, nove straightly to object
17
kg mean aoan_actionsss,
rel_poo trel_pos_norm (.
18
kg nean actoonsttonll 0,-11 11. Irel pos nom >- 0.03) (1 + g.sttaee 6 Irell pos__rrmm 0.03). D1
19
-pict-tnd-Placs, X61, nove straightly grip pos) Irel gos norn 0,03).
goal
21
kg sean Is, reest geal current
22
Inean_attion|l;.ly t-1. pree__oo__oomm (io ( g_state [rol_pos_tomm >n 0.03) (() (1,0)
24
std tb.onss//bstttt nun, att)or_dim) +
A.2.3 BASELINE ALGORITHMS
BC. Since KG1 and KG2 are not in conflict with each another, we can combine these two knowledge mappings (KG1+KG2). The results of BC is obtained by applying KG1+KG2 in both environments.
SAC-BC. The external knowledge policy KG1+KG2 can generate demonstrations to guide an agent. We use this policy collect demonstration data with 2 10t steps for Reach and 10t steps for Pick-And-Place. Then we follow method proposed Nair al. 2018) to train an agent. The demonstration batch size is set to be 128 for both environments. The coeflicient of the BC loss is set to be 10 -3 for both environments. Note that we only apply this loss to the mean of the policy's 's output, and the standard deviation is not learned with any supervised signal.
A.2.4 MODEL ARCHITECTURE
Each external knowledge embedding is generated by PyTorch module, nn. Embedding, with dy for both Reach and Pick-And And-Pl The inner actor network, d.n. is an MLP for both tasks, din in Reach with two hidden layers and a hidden size of 64 units, and that in Pick-And And-Place is with three hidden layers and hidden size of 512 units. The critic network, & is with the same architecture as that of pin. The inner key network, dx. is MI.P for both tasks. Different from the inner key network in MiniGrid environments, die
16

---

Under review as a conference paper at ICLR 2023
OpenAl robotic enviromments does not share layers with den. dx in Reach is with hidden layer and hidden size of units, and that Pick-And-Place with hidden layers and hidden size of 64 units. The query network, ou. is with the same architecture as that Φh. All hidden layers in Oiny Per Uk. and Φu are followed by ReLU activation.
A.2.5 HYPERPARAMETERS
Each training trial in Section 5.2 is run with 5 different random seeds, and each testing trial is run with 100 different random initializations. For the underlying SAC algorithm in Section [5.2] we follow the implementation Mtable-Baselines) (SB3) Rafhnn et al. (202T). The training timesteps for Reach and Pick-And-Place are 20K and 1M respectively. The learning rates are I0 and x 10 respectively. The batch sizes are 256 and 2048 respectively. The sizes of the replay buffer are both 1M. The discount factors y are both 0.95, The temperature of entropy is adjusted automatically in both environments described in Haarnoja al. 2018).
17


Reviews:
[{"review_id": "fv_zQ0Fml0", "reviewer": "Reviewer_ce5j", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an architecture that can leverage external knowledge to make better decisions in RL problems. The architecture implements an attention mechanism that dynamically chooses to attend to an internal knowledge mapping or external ones. It is trained end-to-end via the Gumbel-softmax trick. Experiments are conducted on grid-world and robotic tasks. ", "summary_of_the_review": "The merits of the paper are marginal. I do think the idea presented would be significant for the community. I am learning towards rejecting the paper. \n\n=====After Rebutal=====\n\nI have read the authors' response and decided to keep my current score. My main point of rejection is that the paper over-claims the generality of their methods while not having experiments to back up and avoiding comparing with related work. The definition of knowledge, which can be \"commonsense, suggestion, guideline, principle, specification, rule, or restriction, described as a knowledge mapping\" is very vague without defining what the mapping actual is (e.g. how is that policy related to the optimal policy?). I suggest the authors revise the scope of their claims and provide a more specific specification of their setting. ", "strength_and_weaknesses": "Strengths:\n* Incorporating external knowledge into RL policy is an important problem. \n* The method outperforms proposed baselines.\n* The paper is easy to read.\n\nWeaknesses:\n* The experimental settings and results do not support claims made by the authors. First, the author \"knowledge\" as \"*can be any commonsense, suggestion, guideline, principle, specification, rule, restriction, or previously learned knowledge described in formal or informal language*\", but the knowledge in the experiments is simply commands or high-level options. The authors only construct instruction-following tasks rather than knowledge-incorporating tasks, which are much broader in scope and more difficult to solve. \n* The proposed method is not novel. Attentional architecture is the de-facto choice for building instruction-following agents [1]. The authors simply train the agent end-to-end without proposing any specific techniques that help the policy incorporate the knowledge effectively. Knowledge such as constraints or environment specifications may require more special treatments than simple attention mechanisms [2,3]. \n* There are other ways to design the attention mechanism but the authors do not explain why they choose the presented one. They also do not compare with alternative designs in the experiments. For example, instead of mixing the policies, I can mix at internal representation level h = sum(w_i * h_i), where each h_i is the representation of a piece of knowledge. Then I construct a single policy based on the mixed representation. This avoids having to perform the gumbel-softmax trick. \n\n[1] https://arxiv.org/pdf/1711.07280.pdf\n[2] https://arxiv.org/pdf/2010.05150.pdf\n[3] https://arxiv.org/pdf/2101.07393.pdf\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ebjjzwUoBG", "reviewer": "Reviewer_uBBQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces an agent architecture intended to facilitate the reuse of existing knowledge, called ‘knowledge grounded RL’. The knowledge grounding consists of a dictionary with randomly initialized learnable keys paired with hard-coded policies (these could possibly also be pre-trained, in principle) which is available to the agent. There is one additional policy in the dictionary which is not hard-coded or pre-trained, but trained online. Action selection happens by attending over this dictionary.\n\nThe proposed architecture is evaluated on six MiniGrid and two robotics tasks, with a specific set of hard-coded policies for each environment. With the right set of pre-baked policies, the proposed architecture outperforms the chosen baselines on both environments.\n", "summary_of_the_review": "The idea of re-using existing knowledge, or more specifically, of policies trained or defined previous to the agent's initialization, has merit. However, the implementation and evaluation of that idea in this paper falls well short of the standard for novelty and quality for an academic contribution. The method relies too much on hand-crafted solutions, and the evaluation tasks are too simple and too few and unvaried. I recommend rejection.", "strength_and_weaknesses": "- Weakness: experimental evaluation. The MiniGrid environment is clearly a toy one, which might be suited for a proof of concept, but is not sufficient for a serious evaluation. The two robotics tasks are quite elementary as well; more complex tasks exist, and would be very natural applications for the proposed method - a very simple example being stacking blocks.\n- Weakness: the method as it is used in the paper relies heavily on hand-crafted pre-baked policies. The paper does not show how agents learn policies that can be transferred to agents learning other tasks.\n- Strength: the related literature section is fairly comprehensive and covers a good number of related topics.\n- Strength: the proposed method is explained clearly.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aUroUA9HQR", "reviewer": "Reviewer_quir", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper defines a knowledge-ground reinforcement learning (KGRL) problem that an agent learns to follow external guidelines and develop its own policy. It also provides a realization of this problem by using an attention-based actor model which can switch between either a learnable internal policy or external knowledge. Experiments on MiniGrid and OpenAI-Robotics show the KGRL agent's effectiveness.", "summary_of_the_review": "I like the main idea behind this work, and the proposed actor model architecture looks reasonable. However, since many important details of the proposed method are missing, and there is no source code provided to check how the proposed model is implemented, I thus can not recommend acceptance of this work in its current form. I strongly encourage the authors to re-organize the manuscript's content and re-write the whole paper for better understanding by the audiences.", "strength_and_weaknesses": "Strength:\n1. The idea that agents learn to follow external guidelines and develop their own policies makes sense.\n2. The related work description is very detailed.\n\nWeaknesses:\n1. As mentioned in Section 4.1, \"We learn a key for each external knowledge mapping...\", how to learn these keys? I did not find any descriptions.\n2. In Figure 1, $k_in$ is generated by the Inner Actor, which can be learned during the interaction. However, there is no input about $k_gi$. Is it missing? Or please give some more description about my misunderstanding.\n3. In the paper, PPO and SAC are used as the baseline algorithm respectively, and I hope that more baseline algorithms can be combined to illustrate the generality.\n4. At the beginning of Section 5, this paper claims to answer the question \"(1) Can our KGRL agent learn effective new knowledge by referring to an arbitrary set of external guidance?\". Does the term 'new knowledge' simply refer to the stringing together of old knowledge? If so, the use of the word \"new\" is not appropriate, and if not, could you please give some experimental examples?\n5. Since Atari games often appear in reinforcement learning algorithm articles as a basic verification environment, it is hoped that this paper can also do a brief algorithm verification in this type of environment.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UFbsmSNvcW", "reviewer": "Reviewer_c6pa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Knowledge-grounded RL (KGRL) for the purpose of incorporating, reusing, recomposing and generalizing external knowledge in RL tasks. Taking a unified form of external knowledge as an external knowledge policy, this work proposes an actor model that adaptively weighs or activates different external knowledge policies and a learning inner policy. This is realized by a typical self-attention mechanism, with a state-dependent query, inner/external policy-dependent keys. Query, key embeddings and the inner policy are all learned in an end-to-end fashion from standard RL policy signals/gradients. The proposed approach is empirical evaluated in MiniGrid and OpenAI-Robotics benchmark, demonstrating the supriority of the proposed in learning efficiency and generalization ability, given a few pre-defined external knowledge policies.", "summary_of_the_review": "According to my detailed review above, I think this paper is clearly below the acceptance threshold mainly due to the improper claims on knowledge, the lack of novelty of the proposed method and insufficient experimental evaluation.", "strength_and_weaknesses": "$\\textbf{Strengths:}$\n+ The writing and presentation of the proposed method is almost clear.\n+ The related work includes multiple related domains.\n+ The experiments are conducted from multiple aspects.\n\n&nbsp;\n\n$\\textbf{Weaknesses:}$\n- The major issue is on the central notion of this paper, i.e., Knowledge or KGRL. I think it is improper and kind of misleading to use the notion of knowledge through the paper. After reading the paper, I recommend the authors to use the notion e.g., Policy Reuse/Transfer, which is more accurate. \n  - The authors emphasize many times in this paper like the claim ‘all knowledge can be arbitrarily recomposed, rearranged, and reused anytime in the learning and inference stages’. However, the concrete form of external knowledge considered is the external knowledge policy.\n  - In other words, acctually the main content of this work is on how to incorporate, reuse and transfer existing policies in RL. For KGRL at least in my opinion, the key part is how different (or several common) forms of external knowledge are represented, after which how to use the knowledge is to be considered. However, this part is not included in this work and in the experiments, the external knowledge is implemented by the pre-defined policies.\n  - Given the external knowledge policies, the proposed method in Section 4 is not novel to me. The self-attention mechanism and the end-to-end update are straightforward.\n- The experiments provide few impressive results, especially when the pre-defined external knowledge policies are strong and the environments are not challenging.\n- Important related work is missing. For example, Knowledge Guided Policy Network (KoGuN) [1] propose a similar idea, where suboptimal external knowledge is represented by trainable module based on fuzzy logic and then is incorporated into PPO policy network serving as a refine module.\n\n&nbsp;\n\nReference:\n\n[1] Peng Zhang, Jianye Hao, Weixun Wang, Hongyao Tang, Yi Ma, Yihai Duan, Yan Zheng:\nKoGuN. Accelerating Deep Reinforcement Learning via Integrating Human Suboptimal Knowledge. IJCAI 2020\n\n\n\n&nbsp;\n\n$\\textbf{Questions:}$\n\n1) The authors claim “current learning-from-demonstration and knowledge-reuse approaches lack the flexibility to rearrange and recompose different demonstrations or knowledge, so they cannot dynamically adapt to environmental changes”. Do the author mean the ability of zero-shot adaptation or generalization? To my knowledge, policy reuse and policy transfer are well-studied domain in this direction. Can the authors justify this claim?\n2) The authors claim “These two research directions do not enable incorporating external guidance from different sources into learning, so their transferred knowledge is limited to the one learned among similar environments” and “KGRL allows an agent to follow and reuse an arbitrary set of knowledge obtained from different sources in a new task, and these sources can be very different from the new task”. What do the ‘different sources’ mean? External knowledge in different forms? It is confusing to me since this paper does not address how arbitrary forms of external knowledge are transformed exactly into the external knowledge policies. Instead, the external knowledge policies are given in the setting considered in this work.\n3) Using a state-dependent key for the inner policy makes sense to me. However, I recommend the authors to add a KGRL variant without state-dependent key as an additional baseline in the experiments for more complete evaluation and abalation.\n4) I also have concern on the claim “we disentangle the decision component (the query) and the knowledge representations (the keys)”. Indeed, they are separated but they are updated in an entangled form regarding the policy gradients.\n5) I think there is a straightforward alternative to the gumble-softmax way to acting with the inner policy and multiple external policies. Concretely, to sample an action for each policy (with reparameterization) and then to mix them according to a softmax distribution of $w$. Why is this way not considered? Or what is the drawback of this alternative way?\n6) Are the external knowledge policies used in the experimenst stochastic or deterministic?\n7) It is strange to see the very small error bars in Figure 2a and 2b (in fact, I cannot tell if there is a error bar in Figure 2b).\n8) In Figure 2e, why do PPO-KGRL (KG3) and PPO-KGRL (KG1+KG2) perform worse than PPO while PPO-KGRL (all) outperforms PPO?\n9) In Section 5.2, it seems that KG1 is already optimal in Reach task and KG1 + KG2 is optimal in Pick-and-Place. Can the authors provide more explanation on this point? Similarly, KG1 + KG2 + KG3 is also optimal in 5x5 environments since the view range is also 5x5.\n10) The authors mention “indicating that a KGRL agent learns to not only follow the guidance but also mimic its strategy. This imitation process allows an agent to develop an inner policy that outperforms external strategies”. Can the authors provide more explanation on how such imitation happen under KGRL and why imitation can be better than the external policies?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "EkkH4cyofmf", "reviewer": "Reviewer_4vDB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a framework for incorporating background knowledge into reinforcement learning, Knowledge-Grounded RL. The approach uses an embedding-based attention-mechanism model that learns to attend to external and internal knowledge based on observations. The proposed framework is agnostic to the policy training algorithm, and was demonstrated using PPO and SAC algorithms. \n\nThe proposed framework is evaluated on several tasks from two benchmarks: navigation tasks from the MiniGrid environment and pick-and-place manipulation tasks from OpenAI-Robotics benchmark. In both sets of tasks, the proposed framework superior or competitive performance in terms of sample efficiency. ", "summary_of_the_review": "The paper address an important aspect of AI: how to incorporate external knowledge into a learning agent. It proposes an embedding-based actor model that learns to attend to different sources of knowledge and evaluates this on several benchmarks. However, it misses important pieces of the literature on the same topic and doesn't evaluate against SOTA in the space of reinforcement learning with external knowledge. It also misses important benchmarks in the area. ", "strength_and_weaknesses": "Strengths\n\nThe proposed framework demonstrates superior performance in terms sample efficiency on tasks used and does so with different policy training algorithms. \n\nWeaknesses\n\nThe paper misses related work such as Kimura, Daiki, et al. \"Reinforcement learning with external knowledge by using logical neural networks.\" arXiv preprint arXiv:2103.02363 (2021) and Murugesan, Keerthiram, et al. \"Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines.\" AAAI. 2021.\n\nText-based RL benchmarks such as those used in the above works (e.g., Textworld, Zork, etc) are good candidates for this work. The proposed work would be much strengthened it also used these benchmarks and compared to SOTA methods there.\n\nAlthough the proposed framework is described generally, the external knowledge used in the experiments is task specific and could be learned and re-used by existing transfer techniques. Why shouldn't curriculum learning be equally successful in this setting? The experiments seem to be demonstrating a way of recombining/re-using previously learned knowledge, which could be learned internally only using e.g., some curriculum. \n\nThe implementation details are very general. I don't think it would be easy to reproduce the results. For example, details are provided (Appendix) about how to build embeddings for external knowledge but it's not clear how that external knowledge is encoded. This is supposed to be done by the external knowledge mappings but not details are provided for any of the experiments. \n\nLastly, it's not clear what the internal policy is learning. It seems to be that the internal policy is learning to explore the environment and relies on external knowledge for task execution. So, in my opinion, the internal policy is not learning the policy to complete the tasks but the whole model is learning how to use external knowledge without building any internal knowledge for reuse later.\n\nAlso, I'm not sure the learned policy is interpretable. As shown in the experiments, we can see when the agent chooses to use which piece of knowledge but the individual pieces of knowledge are not interpretable in and of themselves, which is generally the issue of interpretability.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject