PAE: Reinforcement Learning from External Knowledge for Efficient Exploration
Authors: Zhe Wu
Haofei Lu
Junliang Xing
You Wu
Renye Yan
Yaozhong Gan
Yuanchun Shi
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement learning
exploration
intrinsic motivation
knowledge
Primary area: reinforcement learning
TL;DR: This paper introduces PAE: Planner-Actor-Evaluator, a novel framework for teaching agents to learn to absorb external knowledge.

Abstract: Human intelligence is adept at absorbing valuable insights from external knowledge.
This capability is equally crucial for artificial intelligence. 
In contrast, classical reinforcement learning agents lack such capabilities and often resort to extensive trial and error to explore the environment. 
This paper introduces $\textbf{PAE}$: $\textbf{P}$lanner-$\textbf{A}$ctor-$\textbf{E}$valuator, a novel framework for teaching agents to $\textit{learn to absorb external knowledge}$. 
PAE integrates the Planner's knowledge-state alignment mechanism, the Actor's mutual information skill control, and the Evaluator's adaptive intrinsic exploration reward to achieve 1) effective cross-modal information fusion, 2) enhanced linkage between knowledge and state, and 3) hierarchical mastery of complex tasks.
Comprehensive experiments across
 11 challenging tasks from the BabyAI and MiniHack environment suites demonstrate PAE's superior exploration efficiency with good interpretability.

Published as a conference paper at ICLR 2024
PAE: REINFORCEMENT LEARNING FROM EXTERNAL KNOWLEDGE FOR EFFICIENT EXPLORATION
Zhe Wu" Haofei Lu Junliang Xing
You Wu Renye Yan' 144, Yaozhong Gan Yuanchun Shi
Yuan Lab, DDeparmeett of Computer Science and Technology. Tsinghua University Nanjjng University, *Peking University {wuzhe, wuyou, yanrenye, Banyoozhong/oiiyuanlaa com, luhf23@mas ils mennilssssnnghha tsinghua..... edu. en, {ilxing, shiye Stsinghussnghu edu.cn
ABSTRACT
Human intelligence adept absorbing valuable insights from external knowl- edge. This capability is equally crucial for artificial intelligence. In contrast, classical reinforcement learning agents lack such capabilities and often resort extensive trial and error to explore the environment. This paper introduces PAE: Planner-Actor Evaluuar,, novel framework for teaching agents to learn to worb external knowledge. PAE integrates the Planner's knowledge-state align- ment mechanism, the Actor's mutual information skill control, and the Evaluator's adaptive intrinsic exploration reward to achieve 1) effective cross-mode cross-modal informa- tion fusion, 2) enhanced linkage between knowledge and state, and 3) hierarchi- cal mastery of complex tasks. Comprehensive experiments across 11 challenging tasks from the BabyAl and MiniHack environment suites demonstrate PAE's su- perior exploration efficiency with good interpretability.
INTRODUCTION
Accepting suggestions from external guidance integral part of human learning. Hu- mans can absorb insightful external knowledge and seamlessly integrate it into their strategies to tackle complex tasks (Council et al., [2000] Mills et al. 2010). By leveraging existing knowledge, human players can quickly adapt completely unfamiliar games after just a few rounds of playing. Their ability to swiftly trans- fer and align existing knowledge with the cur- rent environment is true testament to human intelligence. Building intelligent systems that learn and integrate like humans is ul- timate goal we relentlessly pursue. Currently, reinforcement learning (RL) has demonstrated its potential towards this goal, with remarkable breakthroughs in numerous domains, approach- ing or even surpassing human capabilities on Atari Games (Mnih al., 2013). Go (Silver et al. 2016) and StarCraft (Vinyals et al. 2019).
Action
Gkéel
Partial Observation
Align
Knowledge Fronderr
Knowledge Fronder
Ficaster
Faceeder
Cross Attention
Choss Attention
Planner
Actor
Betuvice
Lidaate
Update
Behavior
Evaluator
Figure 1: The Planner-Acte Actor-AAct---Eae Evaluuto (PAE) framework consists of three key components.
However, currently RL struggles to efficiently capture and integrate the vast amount of expert knowledge already existing the domain. Specifically, RL faces challenges long-term and sparse reward tasks due to lack of guidance, requiring extensive trial and error. Our primary focus in this
"qqual contribution Corresponding Author

---

Published as a conference paper at ICLR 2024
work is teaching agents to leverage external knowledge and approach optimal solutions faster in sparse reward environments.
Although mimicking human capabilities benefits RL, it remains challenging for RL agents to realize the above capabilities (Chowdhury ct al., 2023). Three main challenges arise when training agents to absorb external knowledge to improve their capabilities. (1) Difficulties in knowledge acquisition and representation. Existing approaches utilize Behavioral Cloning (BC) to acquire knowledge by imitating human behaviors directly (Codevilla al., (018)) or adopt Inverse Reinforcement Learn- ing (IRL) to learn reward functions for providing knowledge (Arora & Doshi 2021). However, this necessitates many demonstrations, which are costly label and collect. (2) Obstacles to the integration of external knowledge and internal strategies. Integration requires bridging the gap between the discrete nature of external knowledge and the continuous nature of internal strategies. Two potential directions emerge: one is enabling internal strategies to query external knowledge, but it faces challenges at the retrieval level (Reid al. 2022). The other approach is directly encoding domain knowledge propositional rules into neural networks for warm start, leading to training difficulties and poor generalization (Silva & Gombolay 2021). (3) Challenge of synergistic updat- ing of external knowledge and internal strategies. Large Language Models (LLMs) have recently demonstrated impressive capabilities (Chowdhery et al. [202 Brown et al., 2020), but constant changes in the external world can quickly render them obsolete for time-sensitive tasks (e.g.. news question answering). While some recent studies have attempted to directly map the rich seman- tic knowledge within LLMs to actions (Carta al., 2023 Mezghani et al. 2023 these mappings cannot be updated or come with high update costs.
We propose using natural language as a knowledge source to bypass the difficulties of knowledge ac- quisition and representation. The language contains natural and flexible prior knowledge for efficient exploration and skill acquisition, enabling rapid generalization across tasks. Truly understanding the semantics and focusing on the key information is a good start to integrating external knowledge and internal strategies. To synergize the updating of external knowledge and internal strategies, external knowledge sources and internal strategies should retain the ability to learn and improve over time.
To bridge the gap between RL and human in handling sparse reward tasks, as shown in Figure T propose novel paradigm for guiding agent learning to absorb external knowledge called PAE: Planner-Actuator- Evaluutoo. The Planner is equipped with a state-knowledge alignment mecha- nism. This mechanism enables the Planner access external knowledge sources and retrieve suit- able knowledge that aligns with the current state. This aligned knowledge is progressively provided to the Actor, increasing in complexity. The Actor leverages the state information and the external knowledge provided by the Planner for joint reasoning. It incorporates a cross-attention mecha- nism, allowing the Actor to precisely focus the critical state and external knowledge features. Additionally, the Actor employs discriminative network to reverse-infer the Planner's guidance, strengthening the connection between states and knowledge. The Evaluator calculates intrinsic re- wards based on the quality of external knowledge the Planner provides and the Actor's 's reasoning effectiveness. These intrinsic rewards guide independent updates of the Planner and the Actor
To summarize, our contributions are as follows:
We present a novel framework, PAE, for learning to incorporate external knowledge to achieve better and faster solutions in sparse reward environments. We demonstrate that knowledge instructing significantly improves strategy exploration in RL.
Our PAE framework introduces a circular feedback mechanism. The Evaluator's feedback keeps the Planner aware of the scenario. The external knowledge the Planner provides aligns with the Actor's current capabilities, enabling the Actor to master knowledge from easy to complex.
Our proposed framework produces interpretable sequences of steps, with the Planner pro- viding a comprehensible plan in natural language form.
We evaluated our proposed PAE framework across 1 challenging tasks from the Baby Al (Chevalier- Boisvert et al. 2018) and MiniHack (Samvelyan al., 2021 environment suites. Compared to non-knowledge-grng guided baseline methods, our PAE framework improves performance by an average of 148% and achieves convergence at least 40% faster. Compared to baseline approaches using language guidance, the PAE framework exhibits an average performance improvement of at least 13% and converges least 31 % faster.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
We build upon extensive work in areas related to the study of knowledge and reinforcement learn- ing (RL). Our focus lies in knowledge acquisition and representation, the integration of external knowledge and internal strategies. and exploration in RL.
Knowledge Acquisition and Representation Converting and representing extracted knowledge ef- fectively is the primary challenge in enhancing a reinforcement learning agent's capabilities with external knowledge. Imitation learning and inverse reinforcement learning capture domain knowl- edge by mimicking human actions via state-action pairs, yet these often require laborious human labeling and feedback (Codevilla et al. 2018 Ho & Ermon 2016). Representing knowledge graph (Xu et al., [0200] Zhang et al. 2022). such as a knowledge or scene graph, encounters issues of limited structured data and sparse valid information. [ee et al. (2017) treat documents exter- nal knowledge; the agent must learn to interpret them for task-solving. [Bougie & lchise (2018) use environmental data from object detectors external knowledge, broadening its applicability. In con- trast, our approach employs natural language directly as external knowledge, eliminating the need for preprocessing. This fosters agent exploration guided by inttinii natural interaction.
External Knowledge and Internal Strategy Integration Incorporating external knowledge to en- hance agent decision-making is crucial. Some methods teach Al to use language for querying exter- nal knowledge (Carta et al., 2022 Liu et al., (202)) but grapple with generating sizable, retrievable language queries. Several approaches (Humbird al. 2018 Silva & Gombolay (021]) aim to embed human domain knowledge into neural decision trees yet face the challenge of bridging tree- to-neural network gaps. Our focus linking natural language with policies to guide agent learning- Like Language-condditoonedd RL, approach trains agents to follow instructions in interactive envi- ronments (Luketina et al., 2019). While prior research explored this setting for various tasks in 2D or 3D environments (Colas et al. 2020; Chevalier-Boissee
et al. 2018), we aim to craft comprehen- sive framework that encompasses knowledge selection and representation, the integration of external knowledge and internal strategies, and updating knowledge-guiie and understands key parts of the language for joint reasoning and selects external knowledge to provide knowledge appropriate the current agent's capabilities. This results in enhanced task per- formance, greater adaptability, and improved interpretability. Some more recent work (Carta et al.) 2023 Du et al. (023)) have extracted and employed knowledge from LLMs for decision-making tasks. These approaches often directly map the internal knowledge of LLMs to actions. However, this mapping weakens the policy update potential of reinforcement learning algorithms and might present challenges regarding update costs for üime-sensitive tasks.
strategy. Our approach absorbs
Exploration in RL Exploration stands as a crucial challenge in reinforcement learning, covering e-greedy action selection (Strouse et al., 2021), state counting (Bellemare et al. 2016). curiosity- driven exploration (Schmidhuber) 1991). and intrinsic motivation (Oudeyer et al. (207). Our ap- proach aligns with intrinsic motivation, where novelty fosters access to new states. Other forms of intrinsic motivation include empowerment (Klyubin et al. 2005), promoting agent control over the environment, and goal diversity (Pong et al.) 2020), encouraging increased entropy within the goal distribution. In contrast, we propose intrinsic motivation as a "cyclic feedback mechanism": The Evaluator's 's intrinsic reward prompts the Planner to equip the Actor with knowledge suitable for its current capacity, guiding the Actor to execute tasks aided by external knowledge. Our approach also emphasizes enhancing reinforcement learning through language- guideddg exploration (Mu al.) 2022] Carta et al. (2022). This methodology typically assumes an annotator's presence to generate language descriptions for intrinsic rewards, guiding exploration. We adopt this framework while automating and unifying the entire process, encompassing knowledge representation and selection, integrating external knowledge and intrinsic strategies, and computation of guided intrinsic rewards.
PRELIMINARIES
We formally define the problem formulation of Knowledge-Instructed Reinforcement Learning, We consider augmented Partially Observable Markov Decision Processes (POMDPs) (Kaebbligg et al. 1998). Our augmented POMDP can be described by 7-tuple M (S,A, T, R, f2, O. S is the state space. A donates the action space. Ω is the observation space. is the knowledge set. T S A x S + IR+ is the state transition function. O ::S x A x Ω IR+ is the observation
3

---

Published as a conference paper at ICLR 2024
function. R : 8 x A H R represents the state-action reward. At each time step the agent receives an observation O2 E S2 and a knowledge l E K. Based on these inputs, the agent selects an action a A, and transits to the next state $1+1 S. Via RL, our goal is to find some policy ı : S x K. + A that maximizes the expected return.
We assume that the environment provides knowledge following previous work (Jiang et al. 2019 Mirchandani et al., [021]] Waytowich et al. 2019 Mu et al. (022). We impose no specifie represen- tation requirements and present all knowledge in natural language. Natural language's hierarchical structure and rich semantics profoundly impact an agent's cognitive functions (Colas et al. 2022)
Agents can update internal strategies guided by language's 's generality and abstraction, enhancing comprehension of complex concepts and aligmment with human values. Many modern RL environ- ments include language by default (Mu et al., 2022), like BabyAl
Sreevalier-Boivver et al., 2018).
NetHack (Küttler et al. 2020), MiniHack (Samvelyan et al. (021), text-based games (Côté et al., 2019 Shridhar et al., |020/Urbanek Urbanek et al.] 2019). and most video games. In language-absent envi- ronments, language descriptions relevant to the agent's state can be produced by generative models of language trained on gigantic amounts of text such as GPT (Brown et al. 2020). PaLM (Chowdh- ery al. 2022). and Llama (Touvron et al. 2023
METHOD
We introduce the PAE framework. a novel approach for instructing agents learn to absorb knowl- edge. As shown in Figure 11 PAE consists of three core components: (1) the Planner, policy network that delivers language-based external knowledge guidance, adapting in complexity follow- ing both task features and the Actor's current capabilities; (2) the Actor, mnowledge-oonditoone policy, designed to integrate external knowledge and environment states for joint reasoning; and (3) the Evaluator, module that offers intrinsic rewards to both the Planner and Actor, guiding their strategy updates.
4.1 ALIGNING EXTERNAL KNOWLEDGE AND STATES VIA PLANNER
To enhance guidance, the Planner must progressively select increasingly challenging knowledge for the Actor during the training procedure. This presents two key challenges: 1) correctly under- standing knowledge in natural language and aligning with the current environmental state, and 2) adjusting the complexity of the chosen knowledge based on the Actors' abilities. To achieve this, we consider the Planner as a policy network Tp (kk]so. KC; f) modeled by an MDP: it inputs an ini- tial state so and a set of knowledge IC and selects a piece of knowledge E KC as output. The Planner provides new knowledge whenever new episode starts, the Actor reaches intrinsic goal guided by the knowledge. The Planner receives rewards only when the Actor completes the task using the provided knowledge, aiming maximize cumulative rewards. Figure provides an overview of the Planner.
Encoding. To assist the Planner in better comprehending hidden information within the environ- ment and knowledge, we introduce a state encoder and a knowledge encoder to encode state and knowledge information to embedded features separately. Specifically, for the state encoder, visual state embeddings so E # xe are fed to shape-preserving 2D convolution layer to generate the state feature &u E d. To capture both the content and spatial characteristics of the states, position embeddings Epos are added to šo. which is formulated as:
Su Conv(so) + Epos:
Su E HW XC
So, Epos E HW xd.
where e is the dimension of embedding vectors and de is the channels of feature maps after convo- lution. For the knowledge encoder, each unit of knowledge ky E K. is encoded using a pre-trained language model (e.g. BERT-base) with frozen parameters, yielding the knowledge embeddings ky:
k (6,)) ((2) [(m) Proj(LM((k()) (222 (.,...,,(()))))
k Baxdx
where Proj(-) a linear projection layer and dg is the dimension of knowledge embedding vectors.
'The reason we choose se instead of s, is that the Planner is required to provide macro-level guidance hased on its curriculum, rather than step-hy-step guidance based on the Actor's specific actions at timestep t.
4

---

Published as a conference paper at ICLR 2024
Planner Architecture
opann
State Embedding
door
Full Observation
Cross
Projection
Knowledge Embedding
Atteation
Linear Forward
Knowledge Oracle
External Knowledge
Knewledge--tate Alignment
Figure 2: Overview of the Planner Network
Alignment. To better align external knowledge with the current environment, we utilize the scaled dot-product cross-attention mechanism to enable the Planner to attend between knowledge and states. Specifically, we compute the query Q using the knowledge embeddings k and compute the key and value V using the position rosition-embeddee state feature So. summary, the alignment procedure can be formulated as:
QKT ddz
Ken = Attention(Q, K, = SoftMax
V
where Q kWo, K S. K, V So
where Wo E RRde WK E Rd,xdx and Wv R.d, xdx are learnable projection matrices.
Forwarding. The aross-attention layer produces context vectors k. [k:00 k ..... k that aggregates critical information from both the knowledge set and the state, emphasizing knowledge most relevant to the current state. Then, k2 E Raxd will be linear projected to n 1-dimensional scales and go through softmax layer, aiding the Planner in selecting the appropriate instruction for the Actor
Tp (k) K:; 0) (o..
4.2 TRAINING ACTOR-SPECIFIC INCREMENTAL SKILLS WITH EXTERNAL KNOWLEDGE
The Actor is a knowledge-conditioned policy parameterized as Ta (arl"ss , k; w) Once the Planner provides the Actor with suitable external knowledge, we aim for the Actor's competence in two key areas. First, like the Planner, the Actor must comprehensively grasp knowledge in its natural language form. Second, we want the guiding knowledge to influence the state reached by the Ac- tor, such that different knowledge leads to distinct skills. Accordingly, we seek to strengthen the correlation between knowledge and state.
Encoding, Alignment & Forwarding. To master language comprehension and align knowledge with current observable states, the Actor adopts a three-stage pipeline similar to the Planner: en- coding, alignment, and forward. However, unlike the Planner, the Actor receives Ps, A partially observable state of size h x w, and k, a single piece of knowledge filtered by the Planner, at each time step t. In the Encoding stage, due to the reduced information content of the input state Pst, we remove the shape-preservine beddings. Meanwhile, also utilize the parameter- trraeeter--rozen language model to encode the existing knowledge effectively:
convolution and directly add position information to state em-
PS, Ps; epos
PSS. Ssss Epos
5
Proj([M(M))
E 1xds
During the Alignment stage, we also incorporate the scaled dot-product cross-attention mechanism to achieve the knowledge-state alignment.
QKT Vdl.
P pk. = Attention K, V) = SoftMax
(6)
where Q iWq, K= K V Wv
5

---

Published as a conference paper at ICLR 2024
Actor Architecture
State Embedding
yotaao
St
Conv
Partial Observation
Pre-trained LM
[g(
Knowledge Embedding
Cross Attention
Linear Forward
External Knowledge
kt Knowledge-State Alignment
Figure 3: Overview of the Actor Network
Moving on to the Forward stage, we aim to enhance the Actor's attention to current state informa- tion. To achieve this, employ strided convolution extract state information further. We then concatenate this extracted information with the state-aligned knowledge, passing it through Linear and softmax layer to generate the distribution over actions.
(ax St, k;; w) (o(ooaa)))((
Maximize the mutual information between knowledge and states. We utilize the mutual infor- mation maximization objective to build on the idea that knowledge should dictate the states the agent visits. Our approach maximizes the mutual information [(s; between knowledge and states. The objective can be written as follows:
I (ss k) = H(k) - H((kss)
(lg p({[s)] - Ek)p(t) (ogpp)))))ppk)))
> Ekkp(H),, [log g (k(s) s)- log p(A:)]
As we cannot compute pkk(s) exactly by integrating over all states and their associated knowledge, we approximate p(k(s) using a trainable discriminative network (kks)) Previous work (Fysenbach al. 2018) has proved that maximizing the mutual information between and can translate into maximizing the variational lower bound obtained by replacing p(kls) with (x (as shown in Equation Sp. From this variational lower bound, we give the mutual information rewards TMI used to encourage the Actor to strengthen the connection between state and knowledge:
ri log 90 (kl?ss log p(k).
The discriminative network qs (k((s) is optimized using a cross-entropy loss Cg. denoting as: Co
Σ (K' k) 1og(ooo((as (ksss).. WEK
(10)
In summary, the Planner samples piece of knowledge l: from the logits output by Ty (k]ss, K.; 0) and provides it to the Actor. The Actor employs the conditional network n. (ar|ss,, k; w) to make decisions in the environment guided by this knowledge k. The Actor is motivated explore states that offer higher rewards and have close relationship with k. The discriminative network (kkss) uses the provided by the Planner ground truth and updates through supervised learning bet- ter infer knowledge from the visited states. p(k) is provided by the Planner's policy network T so. KC; 0). More implementation details are shown in Appendix A.3 Algorithm in Ap- pendix I.4.1 summarizes our PAE procedure.
4.3 EVALUATION MECHANISM
Reinforcement learning in long-term and sparse reward tasks often difficult and requires a lot of trial and error. The standard solution to speed up this process is introducing additional reward signals to better guide learning. Incorporating external knowledge offers a more straightforward approach to reward shaping.
For the Actor, the external knowledge from the Planner serves as the missing signal in the envi- ronment. In addition the extrinsic reward for accomplishing the final task, the Actor receives

---

Published as a conference paper at ICLR 2024
additional intrinsic reward of +1 for each subtask completed per the Planner's guidance. Conse- quently, the total rewards for the Actor comprise three terms: environmental extrinsic rewards Tews the maximization of the mutual information term TM and intrinsic rewards Tin
Factor = Tex + Tin (a [0e]s,, k; w) + k)) + aITT,
(11)
where is the indicator function, Ta (a,l"s st: k; w) + denotes that the Actor completed the subtask following external knowledge guidance, as is scaling factor.
For the Planner, we follow the previous approach (Mu et al. 2022; Campero et al. 2020) of using the Actor's task completion as intrinsic reward. This task is neither easy (i.e., the knowledge given by the Planner is too simple) nor impossible (i.e., the knowledge given by the Planner is too complex) for the Actor complete the current task, i.e., an automatic curriculum. Specifically, we evaluate the Planner using the number of steps required for the Actor to follow the guidance proposed by the Planner; if the number of steps required for the Actor follow the guidance is within the threshold t", or if the guidance can not be followed until the end of the episode, the Planner receives a reward of 3, otherwise, the Planner gets a reward of +a:
to if -3
Tplanner Ven + Tinj T'n
(12)
i to or t >
tmaz
For the value of t², use adaptive but heuristic approach, where the threshold increases linearly by one when the Actor successfully follows the guidance ten times.
EXPERIMENTAL EVALUATION
Our experimental evaluation aims to test our central hypothesis: that external knowledge improves the exploration efficiency for RL algorithms in sparse reward environments. We conducted series of experiments organized as follows: 1) In Section 5.1 we quantitatively evaluate the exploratory capabilities of the PAE by comparing to various baseline methods. 2) In Section 5.2 we revealed
the underlying mechanisms of the PAE through ablation experiments. 3) Section we elucidate the interpretability of PAE's strategy by visualizing the agent's learning process at different stages. See Appendix A.5 for additional experimental results. More implementation details of PAE are
shown in Appendix A.3
We evaluated our method across two task types, totaling six environments within the BabyAl environment: Key Corridor tasks (KEYCORRS3R3, KEYCORRS4R3, KEYCORRS5R3) and Obstructed Maze tasks (OBSTRMAZEIDL, OBSTRMAZE2DLHBB OBSTRMAZE Q). The suf- fix signifies the environment's size; larger environment increases the exploration difficulty in the same type of task. To demonstrate PAE'S scalability, we extended our PAE approach to the more challenging MiniHack tasks. MiniHack consists of procedurally generated tasks within roguelike game, offering richer observation space than the BabyAl environment, Additionally, it presents up to 75 dimensions of structured and context-sensitive action space. Our comparisons of PAE with baseline methods encompassed five MiniHack environments: LAVACROSS RING, LAVACROSS-Pon POTION, LAVACROSS-FuL RIVER-MONSTER, and MULTIROOM MULTIROOM-N4-MONTTI N4-MONSTER. See Appendix A.2 for more details.
Environments:
2022 to introduce knowledge. In BabyAI, the Planner We follow the previous work (Mu et al. receives fully observed state and set of 652 instructions from the BabyAl platform. Meanwhile, the Actor gets a 7 x 7 representation of its field of view and a single instruction filtered by the Planner. See Table 3 in Appendix A.. for a list of knowledge provided by Baby AL. In MiniHack, the Planner and Actor have the same field of view. Observations consist of a 21 x 79 matrix of glyph identifiers, -dimennsional vector with the agent's 's statistics (like location and health), and series of 256 charracrr knowledge entries. The Planner accesses multiple knowledge entries, whereas the Actor receives a selectively filtered entry from the Planner. See Table in Appendix A.2 for a list of knowledge provided by MiniHack.
5.1 QUALITATIVE RESULTS: COMPARE TO BASELINES
We compare PAE with six baseline algorithms across three categories to highlight PAE's advan- tages. 1) Vanilla Baseline Algorithms: IMPALA (Espeholt et al. 2018), standard asynchronous
7

---

Published as a conference paper at ICLR 2024
NiyCorrSUR3
KeyCorrSSity
Obstr
0V
156
1PM
101
194
2004
2014
304
104
501
Vieit LrvaCrew--ing
1909
2904
344
1.00
130
100
ObstrMannDDlbb
LavaCrossPodial
18-
62
3M
19A0
1990
2000 2010
07
104
30M
XCM
LavaCressFFut
Maltiraun-Ne-Munee
River Monster
1.4
1.8
PAE (Ours)
T-NowelD
L-AMIGo
AMIGe
ICM
RND
IMPALA
300
1214
Figure 4: Performance of PAE, IMPALA, RND, ICM, AMIGo, L-AMIGo and L-NovelD in eleven environments, with error regions to indicate standard deviation over five random seeds.
actor-critic method, focuses on rapid parallel training and learns only from raw environmental re- wards, without any intrinsic motivation external guidance. 2) Intrinsic Motivation Algorithms: RND (Burda et al., (2188) uses a randomly initialized neural network to compute prediction errors, which serve intrinsic rewards for exploring new states. ICM (Pathak et al., 2017) encourages exploration by predicting the outcomes of actions and generating intrinsic rewards from prediction errors. AMIGo (Campero
al.] 2020) pairs a goal-generating teacher with goal-conditioned student policy, enriching environmental rewards with intrinsic goals. 3) Language-Instructed Instructed Algo- rithms: L-AMIGo (Mu et al. (022) builds on AMIGo's model, using language to spotlight relevant abstract concepts in the environment. L-NovelD (Mu et al. 2022) combines natural language with an intrinsic motivation approach to reward states described in natural language that transitions from low to high novelty. While all the above baselines, except the vanilla ones, incorporate some form of external guidance, L-A.IGo is the most similar to ours. However, introduces language knowledge solely at the level of external guidance without focusing its integration with internal strategies. The full hyperparameter sweep for PAE and all baselines is reported in Appendix A...4
As shown in Figure 4 PAE achieves almost the fastest convergence across all cleven environments. Notably, AMIGo, RND, ICM, and IMPALA fail to converge within the current step limit. Compared to the best performance in non-knowledge-gun guddded baseline methods (AMIGo, ICM, RND), our PAE framework improves performance by an average of 148% and achieves convergence at least 40% faster. Compared baseline approaches using language guidance, the PAE framework exhibits an average performance improvement of 13% and converges at least 31 % faster. Tables 6 and 7 in Appendix AA.5 give the quantitative performance of all models in the BabyAI and MiniHack environments, respectively. Furthermore, we observe excellent training stability in PAE, especially after convergence.
5.2 ABLATION EXPERIMENT AND ANALYSIS
To better understand the advantages PAE brings through introducing external knowledge, this see- tion compares the mechanisms for introducing external knowledge in removing or replacing imple- mentations. Full-Model is the full version of PAE. w/o Curriculum, the Planner attaches equal weight to all external knowledge. w/o Planner eliminates the entire Planner's guidance to the Ac- tor. Table Table[[ displays the final mean extrinsic rewards and the number of steps (in millions) needed for each model to converge. Each entry consists of two rows of results, with the top row being the average extrinsic reward at the end of training and the bottom row being the minimal stable steps to attain that reward. Lower bottom row values signify quicker convergence, and "> x" indicates lack of convergence within the maximum training steps "x". Training curves for the ablation study provided in Appendix A.5
8

---

Published as a conference paper at ICLR 2024
Table Comparison of PAE and ablation models.
Key Corridor Tasks
Obstructed Maze Tasks
Model
KEYCORRS3R3
KEYCORRS4R3
KEYCORRS5R3
OBSTRMAZEIDI
OBsTRMAZE2DinB
OBSTRMAZEIQ
Full-Model
0.89 + 0.002
0.92 + 0.005
0.94 0.001
0.93
0.004
0.87 + 0.018
0.89 0.006
6M
30M
90M
6M
150M
150M
w/o Curriculum
0,86
20M
260M
200M
12M
2-00M
250M
0,047
0.00 à 0.000
0,00
0.000
0.00
0,000
0.00
0,000
0.00 4: 0.000
0.00 0,000
Planner
2.2
200M
2-00M
We can that the "w/o Curriculum," the Planner attaches equal weight to all external knowl- edge, performs well in four of the six environments (KEYCORRS3R3, OBSTRMAZEIDL, OB- STRMAZE2DL.HB, OBSTRMAZE the FULL-MODEL in these four environments. 'w/o Curriculum' earned zero rewards in the other two environments (KEYCORRS4R3) structedMaze require more robust exploration to unlock critical states, while tasks like KeyCorridor emphasize easy-to- -o-fffillff completion of the final task. The 'w/o Planner' model earned no external rewards in all six environments. The above results and analysis prove the critical role of introducing external knowledge, especially of suitable difficulty, for exploration sparse reward environments.
but requires at least 54% more training steps compared to
KRYCORRS5R3) We analyze this because tasks like OB-
5.3 INTERPRETABILITY
One advantage of PAE's introduction of language knowledge is that it can provide insights into de- veloping an agent's abilities during training. We first show Figure (a) that PAE has a similar ca- pability to the approach of the automatic curriculum: Planner generates an interpretable curriculum. In the KeyCorridorS3R3 environment, the Planner first provides the Actor with easily achievable external knowledge (open the door) and gradually increases the difficulty as training progresses (go to the door, pick up the key), ultimately converging to the final goal (pick a ball).
1.0
open the
pick the
pick up the ball
0.8
ge
pick the [C]
[C] door
Procentage
0.6
pick up the
[C]
Steps
150
go
[C] key
go to the
go
ICT Key
100
open the
[C] ball
[C] door
0.2
50-
to thk
[C] door
0.0
0
0M
2M
4M
6M
8M
10M
0M
2M
4M
6M
8M
10M
Frames
Frames
Planner, Knowledge curriculum
(b) Actor, Knowledge acquisition
Figure 5: Interpretation of knowledge-instrueee exploration. The knowledge curriculum introduced by the Planner (left) and the Actor's knowledge acquisition process (right) are illustrated in the KeyCorridorS3R3 environment.
In addition to generating a similar curriculum, Figure 5 (b) also visualizes the number of steps required for the Actor to follow each external knowledge, offering insights into this incremental capability acquisition. We can see that the Planner provides increasingly difficult knowledge during the training process, reflected in a gradual increase in the probability of complex knowledge being presented. Conversely, the Actor gradually improves his proficiency in mastering each knowledge fragment during the training process, reflected in a gradual decrease in the required steps.
CONCLUSION
In this work, we propose Planner-Actor-Evaleee (PAE), knowledge-instructed reinforcement learning framework for efficient exploration in sparse reward enviromments. PAE has several desir- able properties: it achieves alignment and joint inference of external knowledge and internal agent states and emerges as an interpretable curriculum. Moreover, PAE is algorithm agnostic, making compatible with any deep reinforcement learning algorithm.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was partly supported by the Natural Science Foundation of China under Grant No. 62222606 and 62076238.
REFERENCES
Saurabh Arora and Prashant Doshi. survey of inverse reinforcement learning: Challenges, meth- ods and progress. Artificial Intelligence, 297:103500, 2021.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Pro- cessing Systems, 29, 2016.
Nicolas Bougie and Ryutaro Ichise. Deep reinforcement learning boosted by external knowledge. In Proceedings of the 33rd Annual ACM Symposium Applied Computing, pp. 331-335 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 13:1877-,901, 2020.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 1810. 12894, 2018
Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint arXiv:2006. 12122, 2020.
Thomas Carta, Pierre-Yvel Oudeyer, Olivier Sigaud, and Sylvain Lamprier. Eager: answering questions for automatic reward shaping language-guided rl. Advances in Neural Information Processing Systems, 15:12478-12490 2022.
Asking and
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre vees Oudeyer. Grounding large language models in interactive environments with online reinforcement learning, arXiv preprint arXiv:2302.02662. 2023.
Liting Chen, Lu Wang, Hang Dong, Yali Du, Jie Yan, Fangkai Yang, Shuang Li, Pu Zhao, Si Qin, Saravan Rajmohan, et al. Introspective tips: Large language model for in-context decision mak- ing. arXiv preprint ar0iv:2305.1 1598, 2023.
Maxime Snevalier-Boisverr Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: platform to study the sample efficiency of grounded language learning. arXiv preprint htt.:188000822.. 2018.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204. 023111, 2022.
Tanmoy Chowdhury. Chen Ling, Xuchao Zhang, Xujiang Zhao, Guangji Bai, Jian Pei, Haifeng Chen, and Liang Zhao. Knowledge-enhanced neural machine reasoning: A review. arXiv preprint arXiv:2302.02009. 2023.
Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End- to-end driving via conditional imitation learning. 2018 IEEE International Conference on Robotics and Automation, pp. 4693-4700, IEEE, 2018.
Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Frier, Peter Dominey, and Pierre-Yvel Oudeyer. Language as cognitive tool to imagine goals in curiosity driven exploration. Advances Neural Information Processing Systems, 33:3761-3774, 2020.
Cédric Colas, Tristan Karch, Clément Choulin-Frierr and Pierre-Y Oudeyer. Language and culture internalization for human-like autotelic ai. Nature Machine Intelligence, 1068))006811010 1076, 2022.
10

---

Published as a conference paper at ICLR 2024
Mare-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asni, Mahmoud Adada, et al. Textworld: learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, pp. 41-75.
Springer, 2019.
National Research Council et al. How people learn: Brain, mind, experience, and school: Expanded edition, volume 1. National Academies Press, 2000.
Yuging Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXi::2302.0662) 2023.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted letor-leamer architectures. In International Conference Machine Learning, pp. 1407-1416. PMLR, 2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018
He, Mari Ostendorf, and Xiaodong He. Reinforcement learning with external knowledge and two-stage q-functions for predicting popular reddit threads. arXiv preprint arXiv:1704.06217, 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, 29, 2016.
Kelli D Humbird, Luc Peterson, and Ryan G McClarren. Deep neural network initialization with decision trees. IEEE Transactions on Neural Networks and Learning Systems, 3085) 2018.
Yiding Jiang, Shixiang Shane Gu, Kevin Murphy, and Chelsea Finn. Language as an abstrac- tion for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1-2):99-14, 1998.
Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Empowerment: universal agent-centric measure of control. In IEEE Congress Evolutionary Computation, volume 1, pp. 128-135. IEEE, 2005.
Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. Torchbeast: A pytorch platform for distributed rl. arXiv preprint arXiv: /010:009000055555 2019.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684 2020.
lou-Jen Liu, Xingdi Yuan, Marc-Alexandre Côté, Pierre-Yves Oudeyer, and Alexander Schwing. Asking for knowledge (afk): Training rl agents to query external knowledge using language. In International Conference on Machine Learning, pp. 14073-14093, PMLR, 2022.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette. Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. arXiv preprint 190.::19900099222 2019.
Lina Mezghani, Piotr Bojanowski, Karteek Alahari, and Sainbayar Sukhbaatar. Think before you act: Unified policy for interleaving language reasoning with actions. Workshop on Reincarnat- ing Reinforcement Learning at ICL., 2023.
11

---

Published as a conference paper at ICLR 2024
Candice M Mills, Cristine H Legare, Megan Bills, and Caroline Mejias. Preschoolers questions as a tool to acquire knowledge from different sources. Journal of Cognition and Development, (4):533-560, 2010.
Suvir Mirchandani, Siddharth Karamcheti, and Dorsa Sadigh. Ella: Exploration through learned language abstraction. Advances in Neural Information Processing Systems, 10-29529-29540 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 1312.560122 2013.
Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, and Edward Grefenstette. Improving intrinsic exploration with language abstractions. Advances in Neural Information Processing Systems, 35:33947-33960, 2022.
Pierre Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE Transactions on Evolutionary Computation, 11(2):265- 286, 2007.
Deepak Pathak, Pulkit Agrawal, Alexci A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference Machine Learning, pp. 2778-277777 PMLR, 2017.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: state-covering self-supervised reinforcement learning. International Conference on Ma- chine Learning, pp. 2020.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- environments. arXiv preprint arXi::2002./2292, 2020.
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement
learning? arXiv preprint arXiv:2201. 12122222112222112222222222222222 2022.
Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Kuttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neu- ral controllers. In Proc. of the International Conference Simulation of Adaptive Behavior: From animals to animats, pp. 222 222-227, 1991.
Mohit Shridhar, Xingdi Yuan, Marc-Alexande Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768 2020.
Andrew Silva and Matthew Gombolay. Encoding human domain knowledge to warm start rein- forcement learning. In Proceedings the AAAI Conference Artificial Intelligence, volume 35, pp. 5042-5050, 2021.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484
2016.
DJ Strouse, Kate Baumli, David Warde-Farley, Volodymyr Mnih, and Steven Stenberg Hansen. Learning more skills through optimistic exploration. In International Conference Learning Representations, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
12

---

Published as a conference paper at ICLR 2024
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. Learning to speak and act fantasy text adventure game. arXiv preprint arXhv:1903.03094, 2019.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamneck, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforeement learning. Nature, 575(7782)355--3333 2019.
Nicholas Waytowich, Sean L. Barton, Vernon Lawhern and Garrett Warnell. narration- based reward shaping approach using grounded natural language commands. arXiv preprint arXiv:1911.00497, 2019.
Yunqiu Xu, Meng Fang, Ling Chen, Yali Du, Joey Tianyi Zhou, and Chengqi Zhang. Deep rein- forcement learning with stacked hierarchical attention for text-based games. Advances in Neural Information Processing Systems, 33: 33:16495-16507, 2020.
Yifeng Zhang, Ming Jiang, and Qi Zhao. Query and attention augmentation for knowledge-based explainable reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15576-15585 15585, 2022.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
The supplementary material provides additional results, discussions, and implementation details.
Section A.I
we discuss the limitations of our PAE approach and provide insights on
related work.
In Section
A.2
we detail the BabyAl and MiniHack environments and the chosen testing
tasks.
In Section
A.3
we provide the implementation and training details for the PAE and the
baseline al algorithms.
In Section
A.4
we describe our PAE algorithm and the baseline algorithms.
In Section
A.5
we present additional experimental results and analysis.
A.1 LIMITATIONS AND DISCUSSIONS
Limitations Despite the impressive results of our approach, we acknowledge at least two limitations. Firstly, our framework relies on environments like BabyAl and MiniHack provide knowledge. This dependency means that applying PAE in environments lacking Oracle knowledge is challeng- ing. However, we are fully committed to exploring the use of Large Language Models to provide knowledge. Secondly, our current approach only accepts unimodal knowledge in natural language. More exciting extensions will involve human interaction to provide more natural and multimodal knowledge, which deserves further exploration.
Relationship with LLM-based frameworks LLM-based agents are gaining popularity in solving RL tasks, and very recent studies have shown considerable potential (Carta et al. [023 Chen et al., 2023]. There are currently two main categories of LLM-based agents for becision-mmaing: Fine- tuning LLMs using RL for decision-making and 2. Employing LLMs directly for decision-making as plug-ins. These differ significantly from PAE, which uses external knowledge to enhance RL algorithms. In contrast, recent LLM-based agent studies primarily showcase the abilities of LLMs or expand their capabilities using RI. methods. In our PAE approach, LLMS are a tool to aid agents in understanding the semantics of knowledge.
A.2 ENVIRONMENT AND TASK DETAILS
BabyAI and Minigrid The BabyAI platform enables research in grounded language learning in- volving humans. In BabyAl, agents maximize rewards by completing tasks within limited steps, guided by language instructions. The platform utilizes grid world environment (Minigrid), par- tially observable 2D grid world housing agents and objects (available in colors): boxes, balls, doors, and keys. These entities occupy N x M tiled rooms interconnected by doors, which may be locked or closed. Agents can pick up, drop, and move objects, while doors require color-matching keys for unlocking. As shown in Figure 6 we evaluated our approach in the Baby across six environments spanning two task types: the Key Corridor task KEYCORRS3RS, KEYCORRS4R3, KEYCORRS5R3 and the Obstructed Maze task (OBSTRMAZE IDL, OBSTRMAZE2DLHB. OB- STRM Q). Table Illustrates critical properties of the Minigrid environment. 3 shows the 66 categories of templates provided by BabyAI, totaling 652 pieces of language knowledge.
Key Corridor In the Key Corridor task, the 'S' in the environment name's 's suffix indicates the room's 'ss size, while 'R' signifies the number rows. The agent must retrieve an item behind a locked door. The key, concealed in different room, must be found by the agent exploring the environment. Figure II illustraates the flow chart for the agent to complete the Key Corridor tasks.
Obstructed Maze In the Obstructed Maze task, small blue ball is concealed in corner. With the door locked and blocked by the ball, the key to unlock it is hidden inside a box. The agent must execute a series of precise steps. Typically, this involves opening the box to obtain the correctly colored key, using the key to unlock the door, and subsequently accessing the door to reach the objective. Figure 12 illustraas the flow chart for the agent to complete the Obstructed Maze tasks.
MiniHack The MiniHack environment is a graphical adaptation of NetHack (Küttler et al. 2020), featuring richer observation space compared to BabyAI. It includes more symbols and supports
14

---

Published as a conference paper at ICLR 2024
(a) KeyComidaS3R3
(b) KeyConidxxSAR3
(c) KeyComidorS3R3
(d) ObstrotoodMess-1
(c) ObstmotedMaae 2D1hb
ObstuctedMaae 1Q
Figure 6: Six challenging environments in BabyAl we used to evaluate PAE.
up 75 different actions. Observations comprise 21 79 matrix of glyph identifiers, 21- dimensional vector detailing agent statistics, location and health, and real natural language messages received during gameplay. As shown in Figure we evaluate our approach in five environments of MiniHack, including two task types: navigation tasks (RIVER-MONSTER MONSTER, MULTIROOM- MONSTER) and skill acquisition tasks (LAVACROSS-RING, LAVACROSS-POTUR
LAVACROSS- FULL). Navigation tasks in MiniHack challenge the agent program to navigate various obstacles, like crossing river by maneuvering boulders traversing intricate or randomly generated mazes to reach specific destination. Skill Acquisition tasks leverage the extensive variety of NetHack's objects, monsters, and dungeon features, including their interplay. For instance, to wear a ring, agent must choose the PUTON action, select the ring from the inventory, and decide which hand to wear on. Table 2 details MiniHack's main fcatures, while Table enumerates some real natural language messages received by the agent during gameplay.
Knowledge in BabyAl and MiniHack (1) The knowledge presented in Table 3 for BabyAl and in Table for MiniHack can be regarded target states. For example, in MiniGri,, the message "go to the door" can be interpreted as "(in this state), go to the door." In MiniHack, the message "the o is killed!" be interpreted as "(in this state), kill the o!".. We view these system text descriptions as the required states that the Planner hopes the Actor will achieve after taking a series of actions. Consequently, the Actor must either take the appropriate action to achieve these goals or extract useful strategies from the actions (trajectories) previously taken toward these goals. (2) Much of the knowledge in MiniHack does not directly contribute task completion. Given that MiniHack's knowledge is derived from actual player feedback during gameplay, it naturally includes irrelevant or emotional messages, such as "ouch!" and "never mind." A rough estimate provided here: an agent capable of solving the Lavacross task will encounter approximately 80 messages, of which only 6-10 (8-13% 13%) are necessary for successful trajectory. This requires Planner to semantically understand these environment messages and filter and exclude those useless knowledge.
A.3 IMPLEMENTATION DETAILS
A.3.1 MODEL ARCHITECTURE DETAILS OF PLANNER
This section provides more details of the Planner architecture used in our method. Planner consists of three components, i.e., encoding, alignment, and forwarding of states and knowledge.
Encoding Following BabyAl, the environment observation is symbolic H x W x fully observed representation, with varying according to the environment's 's size. Each cell the H x grid has features indicating the object's type (e.g., boxes, balls, doors, keys), color (from 6 possible
15

---

Published as a conference paper at ICLR 2024
Table 2: Examples of BabyAl and MiniHack environments and their entity labeling.
Illustration
Observation
Other Properties
Action:
Observation: In Baby AI, the agent's observation space is an ego- centric 7x7 grid representation ori- ented in the direction the agent faces. Each cell within this grid has three defining features: object, color, and state. These features identify the object in the cell, its color, and its state (e.g., distinguish- ing between locked and unlocked door).
Turn left, Turn right, Move for- ward, Pick up an object, Drop, Toggle, Done Reward:
A reward of ' 0.9 x (step count / max steps)' is given for success and for fail- ure.
Termination:
The agent picks up the correct object.
Timeout.
Action:
Observation: In MiniHack, the agent's observation consists of 21 79 matrix of glyph identifiers, a 21-dimensional vector with the agent's statistics (like location and health), and series of messages, "message" is the UTF-8 encod- ing of the on-screen message dis- played at the top of the sereen, rep- resented 256-dimensional vec- tor. Each glyph represents com- pletely unique entity, with integers ranging from to MAX MAX_GLYPH (5991).
MiniHack contains 75 actions, including movement com- mands, managing inventory, casting spells, chopping wands, and more.
Reward & Termination:
MiniHack's RewardManager allows you to specify one or more events that can yield varying (positive or negative) rewards. also enables you to control which subsets of events are sufficient or required for episode termination.
choices), and its state (like open or closed for doors). The Planner embeds these features into type/color/state embeddings with dimensions of 5, 3, and resulting in visual embedding so of size HW x 10. To process 8o, the Planner employs a 4-layer shape-preserving convolution neural network interleaved with the Exponential Linear Units, where each convolution layer has 16 filters with a size of 3timess, stride of 1 and padding of 1. The convolution neural network's 's output & provides an embedding layer sized HW x 16. The Planner integrates randomly initialized, trainable position embedding Epos with Conv(sp) to encapsulate the state's 's content and spatial features and make preparations for alignment.
Meanwhile, the Planner utilized a pre-trained BERT model with frozen parameters to understand the semanties and encode knowledge. Specifically, we use the vector output of the encoder (in this case, BERT) in the [CLS) position for the sentence embedding. This model encoded a set of natural language instructions, supplied by the Oracle, into instruction embeddings with dimensions of n x 768. Subsequently, linear projection layer transformed these embeddings to produce outputs with dimensions of 16, denoting as k.
To integrate the two types of input embeddings. &o E # k E Rax16 we employ the scaled dot-product cross-attention mechanism. The query Q is computed using kWQ, and the key and value are derived from š; WK and & Wy. respectively.
Alignment
Forward After aligning knowledge and environment states, the cross-attention layer yields out- 16 context vector capturing the correlation between the knowledge set and
put, k which is the current state. We project this context vector onto n-dimensional logits through a linear layer and finally get the most suitable external knowledge for the Actor after softmax sampling.
16

---

Published as a conference paper at ICLR 2024
Table 3: All knowledge is provided by BabyAl and covers 66 template categories totaling 652
entries. [C] is one of 6 possible colors: green, grey, yellow, blue, purple, and red.
Ghe <Oiject>
Open <Ojject>
Pick Up the <Otjet>>
go the ball
to the box go door
pick up the hail
open box open the door open the <C> box open the <C>door
pick up the
go the key
plck the key
go the <C>bell go to the <ox
pick up the +C-ball
pick up <C>
go to the <C>door go to the -ay
pick up <ay
Put the -Ohject>-eed to the <Object>
put beil nest to the ball put beil nest the put the next the door
put tal next to the <C>ball ball put the next to the <C>box
put the
to the <Coooor
put the CC next the <C>
the bail next to the key the nest to the ball
the ball next to the <C>kay
put the <C> next the <
the box next to the CCebll
put
ned to the the bix ned to the
door
put the box nest to the box
put the box nex to the <C>box
key
put the box next to the door
put the box next to the <Codoor
the
next to the Cabbll
put the box mext to the key
put the box next to the <C>key
the - key next to the <C
put the key next to the ball
put the koy next to the <C> ball
put the lay ned to the <Cdour
put the key next
put the key ned the
put the -C- key next to the CC>kay
door put the key next the key
put key next
put the key next the dood
put the key next to the <C>key key
(a) LavaCross-Ring
(b) LavaCross- Potion
(c) LavaCross-Full
L
(d) MultiRoom-N4-Monste
(c) River-Monster
Figure Five challenging environments in MiniHack we used to evaluate PAE.
A.3.2 MODEL ARCHITECTURE DETAILS OF ACTOR
This section details the Actor architecture used in our approach. The Actor consists of policy network and a discriminative network.
The policy network adopts similar network architecture as the Planner, with the difference that the inputs to the policy network at each time step are a partial observation of size 7 x 3 and single piece of knowledge & filtered by the Planner. Due to the reduction of the information content of the input state, we remove the shape-preserving convolution and add the position information directly to the state embedding Fs,, obtaining P&, of size (7 x 7) x 10. For the knowledge selected by the Planner, utilize the same parameter-frozen BERT model to get the knowledge embedding i of dimension 64 after linear projection. We compute the cross attention of P&, and i by using
17

---

Published as a conference paper at ICLR 2024
Table 4: Some natural language messages in MiniHack. "message" is the UTF-8 encoding of the on-screen message displayed at the top of the screen.
River-Monster
MultiR Mote-oooo---Mmaee
LavaCross
the stairs are solidly fixed to the floor:
-in what direction?
-what do want to throw?
-with great effort you move the boulder. -hat was close.
that hurts!
-you would bum to crisp trylng to pick
-it's a wall.
things
it's solid stone.
you see no door there.
what do you want to use or apply?
"you don't 't have anything to zap. youu the o.
you kick at empty space.
-you move over some lava. the flint stone falls down stairs.
you strain muscle.
-you try crawl of the water
this door locked.
the flint stone hits amother object.
-your movements because of your load.
slowed slightly
dumb move! -ouch!
youu see here uncursed flint stone.
-sorr, don't know how to use that.
the is killed!
as you kick the door, it crashes open!
"oou float in the opposite direction.
-you try to move the boulder but in vain.
this door is broken.
there is staircase up here.
-ouu pull free from the o.
you hear the wuiling of the banshee...
you don't 't't have anything to drink.
-you are almost hit by dart. picks up a food ration. -you have little trouble lifing j scroll.
as you kick door, it shatters to pieces!
in what direction?
doorway
no door.
it's wall.
whaa a strange direction!
there nothing here to pick up.
you have little trouble lifting) o compse.
your leg feels better.
"you don't 't have anything
apply.
-nvee mind.
-emile's 's ghost touches you!
aggent's ghost touches you!
k, to generate query Q, P3, for key K and value Q. Then, we used four strided convolution layers 32 state embedding. Finally, we concatenated state
to extract the state information to get embedding and knowledge-state alignment embedding and passed it through one linear layer and softmax layer to get the logit distribution over the seven actions.
The Actor uses separate discriminative network go (A)(ss)) to infer the knowledge provided by the Planner from the learned strategies. Intuitively, the Actor motivated to bridge the connection between knowledge and states and explore states with close relationship with k. The discriminative network go (k)()s)) is optimized using a cross-entropy loss Lg. denoting as:
Lo(Sst,k)
Σ (K' k) log(go((((ss)) KEEK
(13)
Specifically, we employ a same-structure strided convolution network in the policy network to get the state feature Päg and knowledge embeddings k similar to the Planner:
f; = Conv(Ps:)
E RR 1x132
(14)
k,- = Proj(LM((AA)) ((2) ,(?))...,,, E IR nx33
Then dot-product operation between Pš, and knowledge embeddings is performed, followed by softmax layer outputting 46 (k)ss)):
La (k()s)) SoftMax(k, PS.
(15)
Interestingly, whereas the discriminative network was initially derived from the definition of mutual information, we find that the discriminative network can be viewed as predicting and modeling the knowledge distribution of the planner from its perspective to strengthen the connection between state and knowledge.
A.3.3 TRAINING DETAILS
Each model was trained using five independent seeds on system with 112 Intel@ Xcon@ Platinum 8280 cores and 6 Nvidia RTX 3090 GPUs. Run times ranged from 10 hours (for OBSTRUCTED- MAZE 1 DI.) to 100 hours (for the longest KEYCORRIDORSRR3 task).
18

---

Published as a conference paper at ICLR 2024
A.3.4 HYPERPARAMETERS
For PAE, we ran grid search over batch size E {8, 32, 150), unroll length {20, 40, 100, 200), entropy cost for the Actor E {0.0001, 0.000, 0.001], the Actor learning rate {0.0001, 0.0005, 0.001}, the Planner learning rate E f0.0001, 0. 0.0005, 0.001}, entropy cost for the Planner E {0.001, 0.005, 0.01}. Table Shooss the best parameters obtained from the search.
For RND and ICM, we followed previous work (Raileanu & Rocktäschel 2020) and used batch size of 32, unroll length of 100, RMSProp optimizer learning rate of 0.0001 with = 0.01 and momentum 0, intrinsic reward coefficient of 0.1, and entropy coefficient of 0.0005, which were the best values they found using grid searches in the same tasks.
For AMIGo and L-AMIGo, we followed previous work (Campero al. 2020 Mu et al., 2022) and used teacher policy batch size of 32, student policy batch size of 32, teacher grounder batch size of 100, RMSProp optimizer learning rate of 0.0001 with 0.01 and momentum of intrinsic reward coefficient of 1, unroll length of 100, value loss cost of 0.5, entropy cost of 0.0005, which were the best values for the same tasks described their paper.
Table 5: Hyperparameters of PAE adopted in all the experiments.
Parameter
Value
Actor batch size B
32
Planner batch size By
32
Unroll Length Discount
100
0.99
Value loss cost
0.5
Actor entropy cost
0,0005
Planner entropy cost
0.5
Actor mutual information cost ay
0,0001
Actor cross attention head number
Planner cross attention head number Pretrained language model
1
Bert-base
Pretrained language model output dim Actor language embedding dim Planner language embedding dim Planner embedding projection dim
768
64
256
16
Actor learning rate
5e-4
Planner learning rate
5e-4
Adam betas
(0.9, 0.999)
Adam
le-8
Clip gradient norm
40.0
19

---

Published as a conference paper at ICLR 2024
A.4 ALGORITHMS DETAILS
A.4.1 ALGORITHMS DETAILS OF THE PAE
Algorithm [ illustrates the overall rollout process of PAE. We implemented the PAE and repro- duction baseline algorithms using TorchBeas (Küttler et al. 2019), an IMPALA-based PyTorch platform for rapid asynchronous parallel training in reinforcement learning.
Algorithm 1: Asynchronous learning algorithm for PAE
Input: Buffer B, Actor batch 18, Planner batch Bp, Actor batch size B, Planner batch size Bp Initialize: B + 2, B f- 2, Bp + Ø, ( St, k; w) + wus Tp | sq. K.; 0) t 0g
Function INTERACT(Actor Policy: z Planner Policy: z. Buffer: B): {so. P } < Env. Env.Reset(O)
>> So denotes full observation," Pso denotes partial observation
pp (kiol s0: K:; )*))
ksteps 6-
while True do
Or f- Tz (url St, k; w*)
[st+1, "5-1, re, Edone: Sdone } <- Env.Step(a:)
Interact with the environment
kateps + keteps +
if donee then
10
80. Psp] &- Env.Reset(O)
This episode finished
k T. (Kolso, KC; 0*)
ksteps 0
end
14
kidone
then
k T 80,
so, KG 0*)
> The Actor successfully followed the Planner's guidance
Ksteps
0
end
18
Update B
Update Buffer B using newly collected experience
19
end
20 end
21
Function MAIN(:
22
Ta + copy)
Make a parameter-fixed copy of Ta for environment interacting
23
T. f- copp))
Make a parameter-fixed copy of Tp for environment interacting
24
Th < Thread(INTERACT, n u B)
Create a thread for environment interacting
25
while not converged do
Update B B
Sample batch B of size B from Buffer B b Training policy Tr using reinforcement learning
27
Train n. (ag/ l;; w) on Training discriminative network 4s using supervised learning b Update π, with latest model ma
Train qs (k)(st) on IB
29
Update n. +- TT. copy((
Update B. with (so, k. kiteps: Sdone) from Rp
31
if |BI I > В, then
Train 7 (k(s0,, K; on BB,
P Training policy up using reinforcement learning
33
B, + Ø
34
Update Fæ Tp. copy()
Update nे. with latest model Tp
35
end
36
end
37
Th.join))
> Stop interacting with the environment
38 end
20

---

Published as a conference paper at ICLR 2024
A.4.2 DETAILS OF THE BASELINE ALGORITHMS
Below, we describe the implementation details of the various baseline algorithms. IMPALA is an off-policy actor-critic framework. The actor-critic agent maintains a policy TT(a(z) and value function vy (z:). both parameterized by 6. Both the policy and value functions are refined using the actor-critic update rule. IMPALA also incorporates an entropy regularization loss. The update is derived from the gradient of a specific pseudo-loss function:
Cvalue (8)
Σ
(es Ve (x,))²
SET
Cpolicy (0)
Σ
Ps log noe (a)l (ra + yus+1 Vo (xx))
SET
(16)
Liatropy (f)
To (a/x,) log To (a/x*s)
SET
[(0) ge (0) + 9p-rolic (0) ge Limmoop (0).
This decoupled architecture facilitates high throughput. Nonetheless, there can be a bias since the policy generating the trajectory might lag several updates behind the learner's policy when comput- ing the gradient. IMPALA addresses this potential bias by employing the V-trace, which balances the variance-contraction trade-off in these off-policy updates.
RND The main idea of RND is to calculate intrinsic reward based on the prediction error linked to an agent's transfer. RND employs two neural networks: a fixed, randomly initialized target network that defines the prediction task and predictor network trained the agent's ' collected data. RND considers the prediction error from the predictor network, based on features produced by the target network, reward for novel states. We followed AMIGo (Campero et al. 2020)) using a re-implemented version based on TorchBeast (Küttler et al. 2019) provided by (Raileanu & Rocktäschel 2020).
ICM utilizes curiosity as an intrinsic reward, defining it as the agent's error in predicting its own behavior's consequences within a feature space learned through a self-supervised inverse dynamics model. Specifically, ICM encodes states St and st-1 as features os;)) and 6(s++1), respectively, trained to predict a, (i.e., the inverse dynamics model). The forward model takes ((s;) and a as inputs and predicts the feature representation (($+11) for 8t+1. The prediction error within the feature space serves as a curiosity-driven intrinsic reward. We followed AMIGo (Campero et al.) 2020) using re-implemented version of ICM provided by (Raileanu & Rocktäschel] 2020). based on TorchBeast (Küttler et al. 2019).
AMIGo is meta-learning method that automatically learns to self-propose adversarial motivational intrinsic goals. AMIGo consists of two subsystems: a goal-conditioned student policy that outputs the agent's actions in the environment and goal-generating teacher that guides the student's 's train- ing. These two components train in adversarial training, where the student maximizes rewards by reaching the goal quickly possible, and the teacher maximizes rewards by proposing goals that the student can reach but not too quickly. We use the open-source AMIGo (Campero et al., (020) code as a baseline for our PAE method.
L-AMIGo utilizes language as a generalized medium to emphasize relevant environmental abstrac- tions. It is an extension of AMIGo where the teacher presents goals using language instead of coordinates (z, v). The student is a conditional policy, receiving language goals instead of (z, y). Specifically, the student network decomposed into policy and grounding networks. The policy network produces a distribution of goals based on the student's 's state, while the grounding network predicts the probability of achieving the goal.
L-NovelD combines natural language with an intrinsic motivation approach to reward states de- scribed in natural language that transitions from low to high novelty. It is an extension of NovelD, which includes two main terms. The first term measures the novelty of state transitions, encouraging exploration without penalizing revisiting less novel states. The second term rewards the agent only for first-time state visits in an episode.
inw.odiiitinnnniiii geals
21

---

Published as a conference paper at ICLR 2024
A.5
ADDITIONAL RESULTS
A.5.1
QUANTITATIVE RESULTS
Figure 4 shows the training curves of our PAE method alongside the IMPALA, RND, ICM, AMIGo, L-NovelD, and L-AMIGo baseline algorithms. Each model was trained using five independent random seeds. The results reported for each baseline and each environment are the best performance configuration of the policy for that environment. 6 give the quantitative results of these comparison experiments in the BabyAl and MiniHack environments, respectively, and show the average extrinsic reward and the number of steps (in millions) required for each model to converge. Each entry consists of two rows of results, with the top being the average extrinsic reward at the end of training and the bottom row being the minimal stable steps to attain that reward. Lower bottom row values signify quicker convergence, and "> x" indicates lack of convergence within the maximum training steps "x".
Table 6: Comparison of PAE and baseline methods in Baby AI environment
Key Corridor Tasks
Obstructed Maze Tasks
Model
KEYCORRS3R3
KEYCORRS4R3
KEYCORRS5R3
OBSTRMAZEIDI
OBSTRMAZE2DIHe
OBSTRMAZEIQ
PAE (Ours)
0.89 1: 0,002
0.92 1 0,005
0.94 1 0,001
0,93 0.004
0.87 1 0,018
150M
0.89 0,006
150M
0.86 = 0,040
0.90 + 0,011
0.93 + 0,016
0.90
0.030
0.23 + 0.403
0.47 = 0.029
L-AMIGo
160M
20M
>-00M
2-300
L-NoveID
0.63 1: 0.097
0,15 1 0.195
0.55 1: 0.484
0.79 0.255
0.57 &: 0,164
0.60 :: 0,107
>-0M
360M
2-00
>20M
>200
>-00M
0,10 + 0.175
0.18 0.023
AMIGo
220M
>60M
200M
220
>200
>-00
0.04 0.052
0.30
0.00 + 0.000
0.31 0.542
0.00 + 0.000
0.00 0.000
>-20M
>60M
>200
15M
>> -00M
>300M
IMPALA
0.00 =: 0.000
0,00 4: 0,000
0.00 4: 0.000
>000
0.00 0.006
>20
0.00 + 0.000
-000
0.00 = 0.000
>300
RND
0.00 4: 0.000
320
0.00 4. 0,000
>A0
0.00 4. 0,000
>00M
0.00 0.000
3-20
0.00 1: 0,000
>200
0.00 d: 0.000
>300
Table Comparison of PAE and baseline methods in MiniHack environment
Model
LAvACROss-RuE RING
AAVACROSS-POTIOS
LAVACROS--FULL
RIVER-MONSTER
(Ours)
1.00 0,001
0,002
1,0 0.001
0.72
0.13 1 0.018 >20
22M
35M
24M
2-40
L-AMIGo
0.57 + 0.223 340M
0,58 0,214
0.55 0.111 >40M
0.31 0.035
0.09 0.016 >-20
3-10
3-10
L-NovelD
0,88 1 0.197
0.99 0,009 30M
0,98 0.015
0.51 0,122
0.02 0.017 >20
240M
30M
3-0M
0.46 + 0.116
0.54 = 0.048
0.44 + 0.101
0.29 0.026
0.08 0.009 >20M
AMIGo
340M
3-10
>40M
3-40
0.00 + 0.000
0.00 = 0,000
0.00 i 0.000
0.18 = 0.199
0.12 = 0.028
ICM
240
240M
240M
340M
>20M
IMPALA
0.00 + 0.000
0.00 = 0.000
0.00 + 0.000
0.00 0.000
0.00 + 0.000
240M
3-10
240M
3-00
2-00
0.00 : 0.000
0.00 0.000
0.00 0.000
0.00 0,000
0.00 :: 0.000
RND
240
2-0M
2-0M
240M
>20M
A.5.2 ABLATION EXPERIENTIAL RESULTS
Figure illustrates the effect of introducing external knowledge PAE performance. Full-Model the full version of PAE. w/o Curriculum provides randomized guidance to the Actors through external knowledge, while w/o Planner eliminates the Planner's guidance.
22

---

Published as a conference paper at ICLR 2024
KeyCorrS3R3
KeyCorrS4R3
KeyCorrS5R3
1.0
1.0
08
0.8
0.8
Rewand
0.6
0.5
Roward
0.5
0.2
0.2
0.0
0.
0.-
094
5M
10M
15M
20M
10M
10M
30M
0M
50M
100M
150M
200M
Piames
ObstrMazeIDI
ObstrMaze2Dthb
ObstrMaze1Q
1.0
1.0
0.8
Rowaed
0,6
0.2
02
0.0
0.0
00
SM
J0M Frumos
I5M
20M
0M
90M
1008
150M
100M
100M
150M
200M
250M
300M
Frumox
Full-Model (Ours)
Curriculum
Planner
Figure 8: Ablation of PAE's external knowledge.
23

---

Published as a conference paper at ICLR 2024
A.5.3 QUALITATIVE RESULTS
Figure SigureDshowe shows an example of PAE completing task in KEYCORRS5R3, In this example, the Actor first needs to explore the environment to find the key, then open the door of the corresponding color, drop the key, and retrieve the item behind the door. Figure [hlws the flow of the Planner forming an automatic curriculum in KeyCorrS5R3 and the progressive acquisition of skills by the Actor Typically, in the early stages, the Planner provides easy guidance. As training progresses and the Actor's ability increases, the Planner provides more hard guidance. We can see in the trajectory that the Planner and Actor are progressing together and eventually complete the task successfully.
10
Figure 9: Example of PAE in completing task in KEYCORRS5R3. To save space, only sampled one sub-trajectory for display.
Easy
Hard
Go to the door
Open the door
Go to the purple door
Open the grey door
Pick up the key
Pick up the ball
Curriculum
Figure 10: An illustration the training process. The Planner forms automatic curriculum the training progresses, and the Actor progressively masters the skills.
24

---

Published as a conference paper at ICLR 2024
A.6 FLOW CHART
Figure 11 and Figure 2 show flowcharts of the agent completing the Key Corridor task and the Obstructed Maze task, respectively.
Move feeward
Move feeward
Turn night
feeward fiewand
Bigla
fievanl
Turn night
Move foewadd
Opea the Jac
No
Tirn left
Drop
Open Backell Bertunng
Yee
Turn left
Move forward
Yoi
Oo bal
right
On Coroiileal
Fed
Puds De
Exd
Figure 11: Flow chart of an agent completing Key Corridor tasks.
right
Ubio (hol
Drep
Pakkpp Pallboo
Ballho! Dus plac
right
Ope
stad
dell Jeiee
hall
Yoe
Corte the (rale)
lal
Figure 12: Flow chart of an agent completing Obstructed Maze tasks.
25


Reviews:
[{"review_id": "VedTr4mqkJ", "reviewer": "Reviewer_qGp8", "summary": "They propose a novel framework for sparse-reward, long-horizon RL tasks consisting of Planner, Actor and Evaluator. Planner provides guidance/knowledge to the Actor in natural language. Actor is incentivized to explore states aligned with that guidance. Evaluator evaluates if Actor is able to accomplish a sub-task given Planner’s guidance and provides intrinsic rewards to both Planner and Actor. The Actor is given a +1 on completing a sub-task and the Planner is given a reward of +alpha if the Actor completed the sub-task under an adaptive threshold number of steps. Otherwise, Planner gets a -beta. The core idea is that the Planner must generate increasingly harder knowledge/tasks for the Actor based on Actor’s current abilities. They use BabyAI for experiments. Both Planner and Actor also receive external environmental rewards.", "questions": "1. I would like to hear authors thoughts on weaknesses 3, 4 and 5 to be sure of my understanding of the paper's contributions\n2. How did the authors reach the design for evaluator? What challenges if any, do the authors foresee in making the evaluator more general-purpose?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is attempting to bring natural language instruction to curriculum-driven RL by adding in Planner to a modified version of Actor Critic\n2.\tTheir encoders use cross-attention to align instruction knowledge with current environment state embedding to encourage selection of the most relevant knowledge and exploration of relevant states by actor. The actor is also rewarded based on the mutual information between knowledge and states.\n3.\tThe paper is well structured even though the methodology is complex.\n4.\tThe reward structure is intuitive to understand and clearly defined for each component.\n5.\tThey compare against IMPALA, RND, ICM, Amigo, L-Amigo and have consistent improvement as well as good training stability across 6 BabyAI tasks\n6.\tThe ablation studies are thoroughly done and establish the value added by the Planner.\n7.\tThe Planner generated curriculum serves as a window into agent behavior", "weaknesses": "1.\tBabyAI action space is quite limited (6 actions) and simplistic, so are the tasks since they are 2D grid world tasks.\n2.\tAs pointed out by authors, current experiments have a knowledge oracle and a very specific template for knowledge peices. Also, planners can observe the whole world, which may be unrealistic for many real-world set up.\n3.\tCurrent experiments require well-defined sub-tasks as well as detection of sub-task completion. Baby AI tasks are about navigating a grid in a particular way and due to task and world structure, it is easy to detect sub-task completion.\n4.\tThis method would need sub-task and knowledge alignment so as to detect if a particular piece of knowledge lead to completion of a particular sub-task. I don’t see how it can work any other way. This is a very hard thing to obtain.\n5.\tThe evaluator currently is highly customized to BabyAI tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bxBkGzu3dZ", "reviewer": "Reviewer_2pSJ", "summary": "The paper presents PAE (Planner-Actor-Evaluator), a novel framework for incorporating external knowledge in reinforcement learning (RL), with the aim of enabling efficient exploration and skill acquisition in sparse reward environments. The framework is built upon three main components: the Planner, the Actor, and the Evaluator. The Planner aligns knowledge in natural language with states in the environment, the Actor integrates external knowledge with internal strategies, and the Evaluator computes intrinsic rewards to guide independent updates of the Planner and Actor. Experiments on six BabyAI environments demonstrated that PAE significantly outperforms existing methods in terms of exploration efficiency and interpretability.", "questions": "The main questions have been mentioned in the `Weaknesses` section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Originality: The paper proposes a new framework addressing the issue of knowledge acquisition, integration, and updating in RL. PAE stands out with its ability to use natural language as a knowledge source and to progressively adapt the difficulty of acquired knowledge.\n- Quality: The proposed PAE framework is designed with clear objectives, tackling key challenges that arise when training agents to absorb external knowledge and improve their capabilities.\n- Clarity & Significance: The experiments conducted in multiple challenging environments provide strong evidence of the framework's effectiveness, its generalization across tasks, and superiority compared to existing methods. The paper is well-written and provides meaningful insights into the relationship between knowledge and environment states, making PAE a valuable contribution to the RL community.", "weaknesses": "I believe that the main limitation of this paper is that the PAE algorithm relies on a pre-defined knowledge set, although the authors have proactively mentioned this in the limitations section. There are already quite a few LLM-based agents that do not require pre-defined external knowledge and are able to complete various tasks well by relying solely on the LLM's inherent commonsense knowledge [1] or by actively gathering, summarizing, or reflecting on dynamic knowledge throughout the task [2]. However, the experimental section does not compare these methods. Although the authors mention that these methods may have higher training/inference costs, I think that given the current popularity of LLM-based agents, this comparison is essential, and higher training/inference costs are not a compelling reason to avoid it. Otherwise, it is difficult to justify the necessity of pre-defined external knowledge and the rationality of the basic settings of this paper.\n\n---\n[1] Carta, Thomas, et al. \"Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning.\" ICML 2023.\n\n[2] Chen, Liting, et al. \"Introspective Tips: Large Language Model for In-Context Decision Making.\" arXiv preprint arXiv:2305.11598 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "htInD9GyeQ", "reviewer": "Reviewer_rZvG", "summary": "This paper introduces PAE: Planner-Actor-Evaluator, a novel framework for teaching agents to learn to absorb external knowledge. PAE integrates the Planner’s knowledge-state alignment mechanism, the Actor’s mutual information skill control, and the Evaluator’s adaptive intrinsic exploration reward. PAE aims to achieve cross-modal information fusion for the actor’s decision.", "questions": "1.The authors use the cross-attention as the alignment, how about the current alignment work like CLIP and ImageBind? Do the authors consider to use this alignment?\n2.In Figure 4, I think there is unfair to compare with most of these baselines, since they do not have or use the language guidance. The L-AMIGo is the only one considering language. Are there any other similar work that the authors can compare? The current comparison is not very strong to highlight your work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This work teaches agents to leverage external knowledge and approach optimal solutions faster in sparse reward environments.\n2.This work introduces LLM ideas to RL problems, which is very interesting and novel.\n3.It can be applied to the current actor-critic approaches.", "weaknesses": "1.As authors pointed, the current knowledge relies on the specific template, which lacks naturalness and does not fully use the huge ability of the LLM. It is an obvious weakness for this paper.\n2.The current knowledge needs human labeling following the specific template, which is of labor consumption.\n3.I suggest the definition and description of the alignment loss can be placed in the main text, which I finally find in the supplementary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "72Dq5FL8Fc", "reviewer": "Reviewer_umQ7", "summary": "This paper presents a novel exploration framework for sparse reward environments, called Planner-Actor-Evaluator (PAE), which teaches RL agents to learn to absorb external knowledge. In particular, PAE adopts a state-knowledge alignment mechanism to enable a Planner to access external knowledge sources and retrieve suitable knowledge that aligns with the current environmental state. Then the actor leverages both of the state information and the provided external knowledge for joint reasoning. Additionally, an Evaluator supplies intrinsic rewards for both the planner and the actor. Experimental results conducted in the BabyAI environments demonstrate the effectiveness of this innovative approach.", "questions": "- What is the difference in wall-clock running time between PAE and the other baseline models?\n\n- Would it be possible to extract latent knowledge from a collected dataset in a pre-training stage and replace the template-based knowledge in PAE?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper studies an interesting topic of using natural language as external knowledge for RL agents to improve exploration in reward sparse tasks.\n\n- The writing is clear and easy to follow.\n\n- PAE showed strong performance in the experiments.", "weaknesses": "- The PAE model comprises three primary components, each with distinct hyper-parameters. Managing these varied parameters can be challenging in real applications.\n\n- Since it depends on special environments and predefined knowledge template, PAE is not generalizable to a broader range of tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)