In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications
Authors: Borja G. León
Murray Shanahan
Francesco Belardinelli
Venue: ICLR 2022 Poster (2022)
Keywords: Deep Reinforcement Learning
Out-Of-Distribution Generalisation
Temporal Logic

Abstract: We address the problem of building agents whose goal is to learn to execute out-of distribution (OOD) multi-task instructions expressed in temporal logic (TL) by using deep reinforcement learning (DRL). Recent works provided evidence that the agent's neural architecture is a key feature when DRL agents are learning to solve OOD tasks in TL. Yet, the studies on this topic are still in their infancy. In this work, we propose a new deep learning configuration with inductive biases that lead agents to generate latent representations of their current goal, yielding a stronger generalization performance. We use these latent-goal networks within a neuro-symbolic framework that executes multi-task formally-defined instructions and contrast the performance of the proposed neural networks against employing different state-of-the-art (SOTA) architectures when generalizing to unseen instructions in OOD environments. 

Published as a conference paper at ICLR 2022
IN A NUTSHELL, THE HUMAN ASKED FOR THIS: LATENT GOALS FOR FOLLOWING TEMPORAL SPECIFICATIONS
Borja G. León' Murray Shanahan' Francesco Belardinclli 1.2 Department of Computing. Imperial College London, London, United Kingdom
2 IBISC, Université d' EvEy, Evry, France (borjagleon, m. shanahan, srancesco.belarddinn eeneelliii@@@mmiiiemmeee @imperial.ac.uk
ABSTRACT
We address the problem of building agents whose goal is to learn to execute out-of distribution (OOD) multi-task instructions expressed in temporal logic (TL) by using deep reinforcement learning (DRL). Recent works provided evidence that the agent's neural architecture a key feature when DRL agents are learning solve OOD tasks in TL. Yet, the studies this topic are still in their infancy. In this work, we propose a new deep learning configuration with inductive biases that lead agents to generate latent representations of their current goal, yielding stronger generalization performance. We use these latent-goal networks within a neuro-symbolic framework that executes multi-task formally-defing instructions and contrast the performance of the proposed neural networks against employ- ing different state-of-the-art (SOTA) architectures when generalizing unseen instructions in OOD environments.
INTRODUCTION
Building agents that generalize their learning to satisfy new specifications is crucial goal of artificial intelligence (Oh et al., 2017; Bahdanau al., 2019; Hill al., 2021). Deep reinforcement learning (DRL.) holds promise in autonomous agents that tackle complex scenarios (Silver et al., 2017; Samvelyan al., 2019), which motivates ongoing research with DRI. algorithms following human instructions expressed in natural language (Yu et al., 2018; Mao et al., 2019). Unfortunately. generalization in DRL is linked to training autonomous agents with large numbers of instructions requiring to build manually those natural language expressions and their corresponding reward functions, which prevents such methods from scaling efficiently (Lake, 2019; Vaezipoor et al., 2021).
These considerations inspired research agents learning from formally specified instructions (Wen Topcu, 2016; Alshiekh et al., 2018: Simão et al., 2021) as a substitute for natural language. Formal languages (Huth & Ryan, 2004) offer desirable properties such unambiguous semanties and compositional syntax, allowing to automatically generate large amounts of training instructions and their corresponding reward functions. Earlier contributions this area rely on the compositional nature of formal languages, often employing multiple policy networks to execute temporal logic (TL) instructions (Andreas al., 2017: Icarte al., 2018; Kuo et al., 2020). However, these approaches do not scale well with the number of instructions since policy networks are computationally costly resource and, consequently. these earlier studies are restricted relatively small spaces environments that require less computation, e.g., non- non-visual settings. More recent contributions (Léón al., 2020; Vaezipoor et al., 2021) have presented DRI. frameworks that are capable of generalizing to large numbers of out-of-distribution (OOD, i.c., never relying on single policy network. Those latter studies evidence that deep learning architectures within DRL agents key towards the ability of agents to generalize formal instructions.
during training) instructions while
However, network architectures in previous literature have mostly followed a standard configuration, where all the network's 's layers except for those encoding the input (e.g., convolutional layers for images, recurrent layers for text) have equal access to both human instructions and environment's observation. We propose novel configuration help agents generalize better by having

---

Published as a conference paper at ICLR 2022
task-agnostic representation of the current state concatenated to latent goal that is form by processing both observation of the agent and human instruction. As a motivation example, let us consider two scenarios: 1) a robot that is at the center of an empty room with red square at the bottom right corner, 2) a robot at the same position in a room that is identical to the one in (1) except that the red square is now green. Intuitively, we can say that, if we give the instruction "go the red square" in (I), the goal of the robot is the same as if we say "go to the green square" in (2), because the two tasks can be abstracted as "turn to face the bottom right of the room, then move straight". In nutshell, computing the human instruction together with the current state of the environment being the center of the room, with the object asked by the human in the bottom right corner allows to deduce that the optimal policy is the same in both scenarios. Our contributions are listed follows:
We propose new deep learning configuration (the latent-goal architecture) that helps agents to generalize better when solving multi-task instructions in OOD scenarios.
We are first evaluate the performance of multiple state-of-the-art (SOTA) neural networks targeting generalization when following temporal logic instructions.
Through ablation studies, we find that networks generating independent pieces of information within their output are specially benefited from being used within latent-goal architectures.
The remaining of the paper is structured as follows: needed to follow this work. Section 3 presents the formal language used to generate instructions and the neuro-symbolic framework that agents use in our experiments. Then, Section details the proposed network configuration. Section 5 includes the experimental settings, empirical results and ablation studies. Last, Sections 6 and 7 discuss related works and conclusions, respectively.
Section 2 briefly introduces the key concepts
BACKGROUND
We develop agents aimed to execute OOD instructions expressed in TL while navigating in partially observable (p.o.) environments. Specifically, agents operate with a fixed perspective and a limited visual range. Below we introduce the basic concepts used in the rest of the paper.
Reinforcement learning. Our p.o. environment is modelled as a p.o. Markov decision process (POMDP). A POMDP is tuple M A, P. Z, O, where (i) S is the set of states s,.'....; (ii) A is the set of actions a. .'.....; (iii) P A x S + [0. 1] is the (probabilistic) transition function; (iv) R: S A x S - IR is the reward function; (v) Z is the set of observations 2, 2', ,..... function::
(vi) O : S x A x S + Z is the observation function. (vii) E [0, 1) is the discount factor, At each time step i, the agent chooses an action 0r A triggering an update in the current state from 8, to 81+1 E 5 according to P. Then, Rg R(5,, G. 8t+1) provides the reward associated with the transition, and ((s a,, 1t+1) generates a new observation 0++1 for the agent. Intuitively, the goal of the learning agent is to choose the policy π that maximizes the expected sum of discounted rewards: V=(x)) (z)
def E Eiz0 n'rr]. where E [0, 1) is the discount factor (see Sutton & Barto (2018)).
Relational networks. Relational networks (ReINets) a particular kind of neural network whose structure is explicitly designed for reasoning about relations (Kemp & Tenenbaum, 2008). Previous contributions on solving formal instructions with DRL have been mainly focused on multi-layer perceptrons (MLPs) and recurrent networks (Goodfellow et al., 2016). yet RelNets are of particular interest in our context since they hold promise improve generalization in DRL (Santoro al., 2018). Their central principle is to constrain the functional form of the neural network in the relational module so that it captures the core common properties needed to reason about the relations between entities (e.g., objects) and their properties (e.g. color), see Santoro et al. (2017). Recently, relational networks have been enhanced with the inclusion the widely-known key-value attention mechanism (Bahdanau et al., 2015). This method relies on the computation of attention scores that are used to decide in which portions of the input data the layer using this mechanism should focus on.
LEARNING TO SOLVE SATTL INSTRUCTIONS
In this section we detail the shared features of our agents. Concretely, Sec. 3.1 introduces the formal language we use to procedurally generate instructions, while Sec. 3.2 details the neuro-symbolic framework that we while testing the different neural network configurations.
2

---

Published as a conference paper at ICLR 2022
3.1 SAFETY-AWARE TASK TEMPORAL LOGIC
We investigate agents learning compositionally instructions in temporal logic. Below, we formally define the language from which we generate instructions to be executed by our agents. Specifically, we extend task temporal logic (TTL), defined in León et al. (2020) to study the ability of neural networks to learn systematically from instructions about reachability goals such as "eventually reach sword". In this work, we evaluate agents that handle safety constrains as well, e.g. "walk on soil or stone until you reached a sword". Thus, we extend TTL to safety-awan task temporal logic (SATTL).
Definition (SATTL), Let AP be a set of propositional atoms. The sets of literals atomic tasks o, and temporal formulas T in SATTL are inductively defined as follows:
[ :== +p - IVI
a : 1U1
T = a TUT
Literals I are positive (+-) or negative (-p) propositional atoms, or disjunctions thereof. Atomic tasks are obtained by applying the temporal operator until (U) to pairs of literals. An atom a WU' is read as it is the case that until I' holds. Temporal formulas T are built from atomic tasks by using sequential composition (t) and non-deterministic choice (U). Note that U and are different operators. We use an explicit positive operator (+) so that both positive and negative tasks have the same length. This is beneficial for learning negative literals (-p) when instructions are given visually, as highlighted in (León et al., 2020). The temporal operators eventually and always D can be respectively defined as ol = truel'l and OI = end, where end is a particular atom true only at the end of the episode. As TTL, SATTL is interpreted over finite traces X, which in this context refers to finite sequences of states and actions. We denote time steps, i.e., instants, on the trace as {[j], for 0 S j [*], where [>] is the length of the trace. While, A[i, is the (sub)trace between instants i and j. A model is a tuple N = (M, where M is a POMDP, and L S + 24' is a labelling of states in S with atoms in AP.
Definition 2 (Satisfaction). Let N' be a model and X finite path. We define the satisfaction relation for literals I, atomic tasks o, and temporal formulas T on path X as follows:
(N, A) = +p
iff
E L(NO)
(N.)) -p
iff
P L(A(0) (N, F or (W.))
(N. = LIVE
iff
(V,)) = -UU
iff
for some 0 S j < (A], (N, IAD FV' for every 1. (0,
(N, - Fl
(W,)) = T:T"
iff
for some 0 < < [A], (N, (V,10.5)) - T and (N, Aij + 1,1)) } T"
(W.A) = TUT
iff
(W, F or I
Intuitively, by Def. 2 an atomic task a = C.U ga, is satisfied if the "safety" condition ca remains true until goal 9. is fulfilled. In the context of this work we not interested in strict safety warranties but in agents trying to reach a goal while aiming to minimize the violation of additional conditions. Thus, we may say our agent has satisfied a even though it has not fulfilled c, at every time step before Ja- Formally. in those cases we consider traces that are not starting at the beginning of the episode, but from the state after the last violation. The agent is penalised for this behaviour through the reward function. Examples of tasks we use are 01 grass U (++axx V +sword) "avoid grass until you reach an axe or sword" and 022 = (+soil V + mud] axe "move through soil or mud until reaching axe". It is not difficult to prove that SATTL is fragment of the well-known Linear-time Temporal Logic over finite traces (LTL,) (De Giacomo & Vardi, 2013). We provide a translation of SATTL into LTL, and the corresponding proof of truth preservation in Appendix F.
3.2 NEURO-SYMBOLEE AGENTS
We extend the neuro-symbolic framework from León et al. (2020), intended for DRL agents solving unseen TTL instructions, and adapted it to work with SATTL. In this section, we provide details about this framework, that entails symbolic and neural module. Intuitively, given a formal instruction T in SATTL, the symbolic module (SM) decomposes into a sequence of atomic tasks a to be solved sequentially by the neural module (NM), which is instantiated DRL algorithm.
3

---

Published as a conference paper at ICLR 2022
Recurrent layer
Task
62-1
Central module
Recurrent layer 2
Recurrent layer
Task
Obs
Encoder
=(me)
Information battleneck
Central module
Recurrent
layer
Obe
Encoder
Lg
((1e)
Decision making
Central module 2
L
Recurrent layer
Decision making
Neural network architectures. Left: standard architecture from previous literature. The central module can be either culne-connected or relational layers. Outputs т T(a,) and V2 refer to the actor and critic respectively (Mnih et al., 2016). Right: proposed latent-goal architecture.
Figure 1:
Symbolic module. Formally, the first component of the SM is the extractor E, which transforms the complex formula T into a list consisting of all the sequences of atomic tasks a that satisfy T. As it is common in the literature (Kuo et al., 2020; Vaezipoor et al., 2021), the SM have access to an internal labelling function Lx Z + 2AP where LT. is the restriction L to the observation space and maps the observation into the set of atoms (e.g, reached sword is atom). Intuitively the labelling function acts as an event detector that fires when the propositions in AP hold in the environment. The second functionality is the progression function P. which given the output of Lr and the current list K, selects the next task a for the NM and updates IC. Once a is selected, the NM follows the given instruction until is satisfied of the SM is to evaluate the fulfillment of the task based on the signals from Lz. From the outputs of Lz, the SM generates a reward function R. which provides the reward signal for the NM. Note that from the NM perspective, this signal is the standard reward signal of a POMDP but in our case associated to the instruction-. Given task a, R is defined that the agent receives a positive reward (+1) 1) if the goal condition becomes true (L.(2+)) ga). strong penalisation (-1) 1) if the agent neither satisfying the goal nor the safety condition (Z+) # Ga or Ca), and small penalisation (-0.05) if none of the previous is true. The latter penalisation is meant encourage the agent to reach the goal. The pseudocode
until the time horizon is reached, while the role
the SM and its functions & and P are included in Appendix A. Since the inner functioning of our symbolic module is analogous to the one in León et al. (2020), where authors show how the SM can process complex formulas T given a NM that has learnt to execute tasks a, devote the remaining sections to the role of the deep learning architecture within the NM when solving atomic tasks, which are the main scope of this work (see end of Sec. 3.1 and Figure for examples of atomic tasks).
Neural module. The NM consists of DRL algorithm interacting with the environment to maximize the discounted cumulative reward from R. All our agents use the same A2C algorithm, a synchronous version of A3C (Mnih et al., 2016). To facilitate fair comparison, all the deep learning architectures use the same visual encoder and output layers for the respective benchmarks. Appendix provides further detail about these features. As anticipated, the core of our approach is the deep learning architecture, detailed below.
4 LATENT-GOAL ARCHITECTURES
We presented the language employed to generate instructions (Sec 3.1) and the neuro-symbolic framework that presenting a new neural network configuration that improves OOD generalization when executing novel multi-task instructions.
agents (Sec 3.2). Now move towards the main focus of this work,
Literature about autonomous agents with ceneralizationnsppciallii architectures such as relational deep layers (Shanahan et al., 2020) or modular hierarchical networks (Mittal al., 2020), commonly
4

---

Published as a conference paper at ICLR 2022
Visual Instruction
000
rreah orange lava reach grey key reach forest key
Figure 2: Left: a 7x7 Minecraft map (left) and its corresponding extended observation (right) with the visual instruction "move through either soil or stone until reaching an axe or a sword". Tasks are specified the top of the extended observation by depicting objects and SATTL operators. Each tile has a resolution of 9x9 pixels. Right: a 22x22 map in MiniGrid where the agent needs to "avoid orange lava until reaching a gray or dark-green key". Instructions are given through a text channel and each tile has 8x8x3 pixels. Highlighted cells indicate the field of vision of the agent.
follows a standard model architecture set-up where the output of a visual encoder is passed through central module (CM), which our context is either relational layer or an MLP, followed by recurrent layer. Unfortunately, deep learning agents struggle to generalize instructions compositionally (Lake, 2019), even when generalizing between task-similar instructions.
Consequently, we propose architecture configuration that induces agents to compute both human instructions and sensory inputs in a dedicated channel, generating a latent representation of the current goal. We call this type of deep learning model latent-goal architecture. Figure right shows our proposed configuration when instructions are given as text; while Figure left illustrates the standard architecture from previous literature for comparison (Hill et al., 2020; Küttler et al., 2020: Vaezipoor et al., 2021). In our configuration, the key feature that the streams Ly and L.. Ly refers to the latent goal, i.e., representation of the agent's 's current goal given the state and the human instructions, and T., is vector that contains information about all sensory input processed by the agent except for the instruction. For a latent-goal architecture to generalize effectively there are two central design properties: i) L. should be task-agnostic, i.c., the layers computing L. should have no access what the current task ii) L, should be passed through an information bottleneck, e.g., a low- dow-diiensional colly-connected layer (FC). Condition i) forces the layers computing L, to generate representations of the states as generic as possible. By granting no access to the current task, these layers cannot discern which elements provide penalisation or rewards, encapsulating information in fashion that facilitates generalization when executing novel instructions. Condition ii) ensures that decision-makin
maainn layers receive two input
layers rely on the generic representation from L, to extract most of the data from the environment, while the layers processing 4g should focus providing enough information about the goal.
EXPERIMENTS
Here start presenting the experimental settings and continue with the main experimental results that highlighting the benefits of latent-goal architectures. We finish with ablation studies to better understand the inner working of the novel configuration.
Empirical settings. We use two benchmarks. One is a Minecraft-inspired environment Minecraft for short - widely used in the RL-TL literature (Andreas et al., 2017: Vaezipoor et al., 202 l). Particularly, we use the variant from León et al. (2020) that works with procedural maps and instructions. In this setting, instructions are visually given as part of the observation. The second benchmark is MiniGrid (Chevalier- -hhevvlier-Boiseertt et al., 2018), which is also procedural benchmark where instructions are given as text through a separated channel. Agents are trained in maps of random size n. for 7 10), and evaluated in maps of size n 14, 22. Figure 2 illustrates maps from
5

---

Published as a conference paper at ICLR 2022
MiniGrid-Test
Minecraft- Test
35
Nomatted
fr 16
7V7
14b34
22x22
Mepaic
747
1601
2b22
(ours)
MLP
MLP Y(ours) PrediNet
RN [(urs)
Muur Preditar"jpun)
MHA
[ouun
B Prediket
MHA
IN [jun))
Pred
Figure 3: Results with OOD instructions in 500 maps (per size) of different dimensions. Sizes 14 and 22 are OOD. MC refers Minecraft whereas MG to MiniGrid. Results show the average reward and standard deviation from 10 independent runs (i.r.). Values are normalized so that 100 refers to the highest performance achieved by the best run globally in maps of the given size and benchmark.
both benchmarks. Instructions are SATTL tasks (see 01 or 02 in Sec. 3.1 for examples), We evaluate generalization to unseen instructions, thus results under label "Train", refer to performance with training instructions, while "Test" to OOD instructions. In MiniGrid, where agents require grounding words to objects, OOD specifications are formed with combinations of symbols and objects that were explicitly excluded from training, e.g., if an OOD instructions has the safety condition "avoid orange lava" it means that both color "orange" and shape "lava" where only present training instructions of the type "reach orange lava". In Minecraft, where tasks are depicted with objects themselves, OOD instructions refer to tasks with zero-shot, i.e., never seen, objects. Since specifications are suman-given, we assume access the resolution of the visual instructions and select an encoder generating a visual embedding where instructions are independent from the rest the observation. This allows the separation of the task from the rest of the observation, which is beneficial for the latent-ge goal architectures that generate task-agnostic streams of data. The separation also facilitates a fair comparison between the text and visual instruction domains. Both benchmarks have least 200K tasks for training and 65K OOD specifications. Further detail is given in Appendix B.
Networks and baselines. All the networks follow either a standard or a latent goal (LG) configura- tion (see Sec. 4). The encoder is convolutional neural network (CNN) (LeCun et al., 2015) while the recurrent layer in the output either a long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997), that use in multi-layered architectures, (BRIM) in the case of hierarchical models (Mittal et al., 2020). In MiniGrid, where instructions are given text, we bidirectional LSTT (Schuster & Paliwal, 1997). We consider four variants for the central modules within the architectures (with detailed information given in Appendix C.3):
bidirectional recurrent independent mechanism
Relation net (RN) from (Santoro et al., 2017), designed dispense FC and LSTM layers with relational reasoning.
Multi-head attention net (MHA) (Zambaldi et al., 2018), which combines relational connec- tions with key-value attention.
PrediNer (Shanahan et al.. 2020), also combining relational and attention operations but specifically designed to form propositional representations of the output.
MLP consisting of single FC layer, which is meant act baseline against the relation- based modules.
Results with multi-layer architectures. Fig. 3 shows the performance of the different multi- layer networks (see Fig. 4 for the reference of random walker). Results that are labelled as CM networks (e.g., "RN"" refer to standard configurations, whose CM uses the network the label. Correspondingly, results with the LG superscript (e.g., RNLG) refer to latent-goal configurations. Overall, we see that the latent-gos architecture improves the performance of all the networks in most scenarios. And is always the best performer in the hardest setting (n = 22), where the best latent- goal model achieves 33% and 22% better performance compared to the best standard architectures in Minecraft and MiniGrid, respectively. Training results and tables with the specific values given in

---

Published as a conference paper at ICLR 2022
MiniGrid-Test
Minecraft-Test
Normalized Seemalized Rewards
80
Normalized Rewards
30
40
10
20
Map size 7x7
14x14
22x22
Map size
7x7
14x14
22x22
Random
BRIM
ResBRIM
(Rurss
Random
BRIM
ResBRIM
(Rurs
Figure 4: Results of 5 i.r. of vanilla BRIM (BRIM), BRIM with residual connections (ResBRIM), and with LGs (BRIMLG).
Appendix D. together with a two-way analysis of variance (ANOVA, independent variables being network architecture in the central modules, and the architecture configuration) that also confirms the statistical impact of the latent-goals. We also observe that the PrediNet is the network that benefits the most from the new configuration (up to 74%) and PrediNet becomes clearly the best architecture. Noticeably, the two most different results with latent-goal architectures come from the networks that combine relational and attention mechanisms (MHA and PrediNet). This point if further examined in the ablation below.
Results with hierarchical architectures. We demonstrate that latent goals improve the perfor- mance of hierarchical architectures. Specifically, in line with the two design conditions highlighted in Sec. 4, we evaluate empirically hierarchical architecture that restricts the amount of information shared between layers of different bierarchy. while providing instruction and observation data to the bottom layer of the architecture and task-agnostic input of the environment the top-level of the hierarchy (see Fig. 8 in the Appendix for illustration). In these experiments we fix the PrediNet as the default network for the different CMs note that the PrediNet achieved the best results in the previous evaluation and replaced the last recurrent layer of the architectures with BRIM. BRIMs are modularized hierarchical networks that hold promise for OOD generalization in RL. A particularly interesting feature of BRIMs for our method is that they employ sparse attention mechanism to communicate between different hierarchies of layers. Specifically, each layer is composed of various modules, and at each time step only the top è modules (according to a key-value attention mechanism) will be able to communicate with modules from different layers, where k is hyperparameter. Thus, when using latent goals with BRIMs remove the FC layer that acted as coormattion-bottleneee above. An illustration of the architectures using BRIMs is given Appendix C.3. We do not perform any hyperparameter optimization with BRIMs and use directly the same configuration from the RL experiments in Mittal et al. (2020). Fig. 4 contrasts the performance when using a vanilla BRIM layer (BRIM) against using a residual connection (He et al., 2016) (ResBRIM) or latent goals (B We see that inducing the latent goals improves the performance of the hierarchical network in all settings. Hence, we see that deep hierarchical architectures can also capitalize on specialized hierarchy levels where bottom layers produce latent representations of the goal while the higher levels reccive task-agnostic information of the state.
5.1 ABLATIONS
Contrasting MHA and PrediNet with their latent-goal counterparts, we see that the particular differ- ences of the PrediNet with respect to the MHA internal functioning benefits the proposed latent-goal architecture. After closer inspection, we see that the most significant difference comes from chan- neling of the input within the PrediNet. This network encourages kind of the semantic separation of the representations that learns, making an element-wise subtraction of the extracted features, an operation not found in the MHA. Additionally, we see that the PrediNet modifies its output to explicitly represent positions of the pair of features selected by each of its heads, also not present in the MHA. Fig. 5 inspects the impact of individually removing these distinct features within the PrediNet and whether the PrediNet is beneficial in both CMs or it better to use an MHA in any of the modules. We find that the element- wise subtraction the key feature within the PrediNet since its
7

---

Published as a conference paper at ICLR 2022
Ablations:
Ablations: PrediNet! variants Test
60
Map-ibo
I0 Preditet"-noooo Predittee"-eoobbb
IN PAMHAA!
MIAAN
ProdiNeet ioPos
Preditee
PNMHAA
MHAPNIO
Figure 5: Ablation studies in Minecraft (5 i.r. per variant). PrediNet and PrediNet variants PrediNet without the element-wise subtraction and the feature coordinates respectively. PNMHALG uses MHA in CM2 while MHAPNLG uses MHA in CM1.
Ablations maps: Bottleneck
Ablations test maps:
99
90
Normalted Rtwwerss
40
40
Map Abz
Map size
14x14
22x22
See-Task-Agnoste
Task-Agnostic
Figure 6: Ablation studies with PrediNet in Minecraft (5 i.r. per variant). Left: Impact different bottleneck sizes. Right: Impact of having Le task-agnostic not.
removal reduces the performance of PrediNet to that of an MHA None of the networks using an MHA in one of the CMs outperform having a PrediNet in both.
Regarding the outlayer of MHA in MC- 14x1,, we believe that receiving instructions not preprocessed by a recurrent layer preventing the MHA overfit as much does in MiniGrid, thus achieving better performance in maps of in three out of four OOD results. As for the reason why the standard PrediNet does not outperform the MHA in the same line that PrediNet-G outperforms MHALD from Shanahan et al. (2020) note that at the core of the PrediNet information is organised into small pieces that are processed in parallel channels limiting the ways these pieces can interact. This pressures the network to learn represen- tations where each separate piece of information has independent meaning and utility. The result is a representation whose component parts amenable recombination. Such feature is highly benefi- cial for generalization, but supposes that the following recurrent layer needs to combine those pieces while producing meaningful output for both actor and critic. As highlighted in Santoro et al. (2017), recurrent layers struggle when relating independent pieces of information. The latent-goal configura- tion alleviates this problem by decoupling the problem of goal abstraction and state representation into separated channels. Consequently, latent- latent-goal architecture helps to exploit the full potential of the highly re-usable outputs produced by PrediNets. Thus, if we substitute the element- wise subtraction with FC layer (as we do with s.) the output is no longer generating independent pieces of information from the parallel channels, which aligns the representations obtained from the PrediNet with the ones generated by an MHA, that employs single channel of information.
size. Still, note that MHALG outperforms MHA
Last, Fig. provides the results of ablation studies regarding conditions (i) and (ii) from Sec. 4. We observe that, for larger bottleneck sizes, generalization performance drops significantly no longer relies on the generic representations from L, since biased information from Lg no longer constrained. That is, larger bottlenecks do not limit the information passed through L, biasing the agent towards the objects used in the training tasks. We see similar degradation in performance when providing information about the task through T.g, confirming that the benefits drawn on the generic
the agent
8

---

Published as a conference paper at ICLR 2022
representations from input that cannot tell on its which objects are beneficial dangerous. Additionally, we perform control experiments in Minecraft to verify that our agents are learning com- positionally from the training instructions, specially with negation, which previous work (Hill et al., 2020; León et al., 2020) have found particularly challenging. We find evidence that with standard ar- chitectures only the MIP and MHA are correctly following the OOD tasks. In the case of architectures inducing latent goals we observe that MLPLC, MHA and PrediNet are able to execute correctly the instructions when applied OOD objects. Results and further discussion is given in Appendix E.
6 RELATED WORK
Applying RL to autonomous agents executing formal specifications has become a thriving area of research. The earliest works combining RL and TL focused on tackling temporally extended goals. naturally described as a non-Markovian reward decision process (NRMDP), by generating equiv- alent extended MDP were RL algorithms can be applied (Bacchus al., 1996). Recent contributions in this line investigate increasingly diverse and expressive languages (Brafman et al., 2018; Camacho et al., 2019) or multi-agent systems (León & Belardinelli, 2020; Hammond et al., 2021). Beyond TL, we find methods such reward machines (a type of finite state machine) (Icarte et al., 2018: Xu et al., 2020; Icarte et al., 2019), or RL-specific formal languages such as SPECTRL (Jothimurugan et al., 2019). Closer to line of work, Kuo et al. (2020) presents novel RL framework to follow OOD combinations known tasks expressed in linear-time temporal logic (LTL) by training multiple networks (one per L.TI. operator and per object). In similar line, Araki et al. (2021) introduces a hierarchical reinforcement learning framework aimed to learn policies that are optimal and com- posable while relying on different neural networks each specialized in one subtask. León al. (2020) provides evidence that DRL agents relying on single neural network can learn compositionally from the training instructions when the convolutional layers are designed according to the environment where agents operate. Vaezipoor et al. (2021) tackles a similar problem in non-visual scenarios by pretraining a neural network to learn to interpret LTL instructions in navigation-bas based environments. Our work advances the state-of-the-arr in agents following formal instructions by presenting a novel network architecture that improves the performance of DRI. algorithms OOD environments.
Due to the multi-modular nature of the proposed configuration, our work also has links to modular frameworks. These configurations hold promise in improving generalization through its natural compositionallty (Jothimurugan et al., 2019). Examples of modular frameworks include DRL agents that combine general control policies, where a global controller is able to combine with different lower-level modules in series of locomotion tasks (Huang al., 2020). Closer to this work Karkus et al. (2020) propose a framework that makes use of a perception and a planner networks to seleets the most suitable human-designed instruction for each time step be forwarded a controller network. This differs from our proposed framework, where latent goals are sparsely communicated and learned from scratch without using any predefined human tasks as embedding of the goal.
7 CONCLUSIONS
By relying on the assumption that human instructions can be separated from the sensory input, we have presented novel deep learning configuration that induces situated agents to generate latent representations of their current goal. Particularly, we have seen that deep learning can draw on architectures where output layers take decisions based on both task-agnostic representations of the environment's 's state and the outputs of a channel that processes observations and human instructions together, we call those outputs latent goals. This configuration yields stronger OOD generalization as long as the latent-goal is forced through a small-enough information bottleneck to the decision- making layers. We have provided evidence of the strong potential this configuration has multiple networks and settings. We believe our findings may have broad impact within the communities working with DRL, OOD generalization and formal methods. Further, this work poses interesting questions about task abstraction and generalization. It also remains open how well DRL agents can generalize when the symbolic part may not provide reliable feedback on task progression. We think that addressing these questions, among many others, will push the boundaries of applicability in RI.
9

---

Published as a conference paper at ICLR 2022
REFERENCES
Mohammed Alshickh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning-Voiie Volume 70, pp. 1756-75. JMLR. org. 2017.
Brandon Araki, Xiao Li, Kiran Vodrahalli, Jonathan DeCastro, Micah Fry, and Daniela Rus. The logical options framework. International Conference of Machine Learning. 2021.
Fahiem Bacchus, Craig Boutilier, and Adam Grove. Rewarding behaviors. In Proceedings of the National Conference on Artificial Intelligence, pp. 1160-1166 1167, 1996.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron C. Courville. Systematic generalization: What is required and can it be learned? In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Ronen Brafman, Giuseppe De Giacomo, and Fabio Patrizi. Ltff/ldf non- non-markovian rewards. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Alberto Camacho, R Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila Mcllraith Ltl and beyond: Formal languages for reward function specification in reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (ICCAI), pp-
0065-6073, 2019.
Maxime Chevalier-Boisvert Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https: / httg//tttuu//e thub. www./aaaimmeeoooooceemaaaeooaeoe hateeeebb//////eee gym seyym-miiigrid. 2018.
Noam Chomsky and David W Lightfoot. Syntactic structures. Walter de Gruyter, 2002.
Giuseppe De Giacomo and Moshe Y Vardi. Linear temporal logic and linear dynamic logic finite traces. Eventy-Thire International Joint Conference on Artificial Intelligence, 2013.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Lewis Hammond, Alessandro Abate, Julian Gutierrez, and Michael Wooldridge. Multi-agent rein- forcement learning with temporal logic specifications. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 583 983-592, 2021.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Végas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1 10.10VP..........900.
Felix Hill, Andrew Lampinen. Rosalia Schneider, Stephen Clark, Matthew Botvinick, James McClelland, and Adam Santoro. Environmental drivers of systematicity and generalization in situated agent. In International Conference on Learning Representations, 2020.
Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. Grounded language learning fast and slow. In International Conference Learning Representations, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8)17355- 1780, 1997. doi: 10. 162/neco. 1997.9.8.777..
10

---

Published as a conference paper at ICLR 2022
Wenlong Huang. Igor Mordatch, and Deepak Pathak. One policy control them all: Shared modular policies for seent-agnottic control. In International Conference on Machine Learning, pp. 4455-4464. PMLR, 2020.
Michael Huth and Mark Ryan. Logic in Computer Science: Modelling and reasoning about systems. Cambridge university press, 2004.
Rodrigo Toro Icarte, Toryn Klassen, Richard Valenzano, and Sheila McIlraith, Using reward machines for high-level task specification and decomposition in reinforcement learning. In International Conference on Machine Learning, pp. 2112-2121, 2018.
Rodrigo Toro Icarte, Ethan Waldie, Toryn Klassen, Rick Valenzano, Margarita Castro, and Sheila Mcllraith. Learning reward machines for partially observable reinforcement learning. In Advances in Neural Information Processing Systems, pp. 15497-15508, 2019.
Kishor Jothimurugan, Rajeev Alur, and Osbert Bastani. A composable specification language for reinforcement learning tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13021-13030, 2019. URL https: ARtttts: http:///coccceeeeeeeeee neurips cc/ http//pppp/////eeeeeee html.
#5aa 7dd22ccdddccccc www.eeeeeaaaaeaeeeeee Abatraaaatttteaaaaaat.... +
Peter Karkus, Mehdi Mirza, Arthur Guez, Andrew Jaegle, Timothy I Lillierap. Lars Buesing. Nicolas Heess, and Theophane Weber. Beyond tabula-rasa: a modular reinforcement learning approach for physically embedded 3d sokoban. arXiv preprint arXiv:2010.01290 2020.
Charles Kemp and Joshua B. Tenenbaum. The discovery of structural form. Proc. Natl. Acad. Sci. USA, 105(31):10887-106) 2008. doi: 10.1073//naas.000233 105.
Yen-Ling Kuo, Boris Katz, and Andrei Barbu. Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas. CoRR, abs/2006.01 10, 2020.
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. (0 Abst html.
http:///pppooedddnnne.n.r.....e neurips cc/ pccppapee///////eee/ccccpp
Brenden M Lake. Compositional generalization through meta pouuence-t--equeeee learning. In Advances in Neural Information Processing Systems, pp. 9788-9798, 2019.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521 521 (7553)):4336 2015.
Borja G. León and Francesco Belardinelli. Extended markov games to learn multiple tasks in multi- agent reinforcement learning. In Giuseppe De Giacomo, Alejandro Catalá, Bistra Dilkina, Michela Milano, Senén Barro, Alberto Bugarín, and Jérôme Lang (eds.), ECAI 2020 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 September 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications, pp. 139-146. IOS Press, 2020. doi: 10.3233/FALA2200086. URL https: www..oooo....2/.0///// FAIA200086.
Borja G León, Murray Shanahan, and Francesco Belartinelli. Systematic generalisation through task temporal logic and deep reinforcement learning. arXiv preprint urXiv:2006.08767, 2020.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The neuro-
symbolic concept leamner Interpreting scenes, words, and sentences from natural supervision. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019.
11

---

Published as a conference paper at ICLR 2022
Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murra Shanahan, Guillaume Lajoie, Michael Mozer, and Yoshua Bengio. Learning to combine top-down and bottom-up up signals in recurrent neural networks with attention modules. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1972-6986. PMLR, 2020.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. sero-shot task generalization with multi-task deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Coanninng-Vllmee 70, pp. 2661-2670. JMLR. org. 2017.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In Proceedings the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2186-2188. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter W. Battaglia, and Tim Lillicrap. A simple neural network module for relational reasoning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pp. 1667-4976,, 2017.
Adam Santoro, Ryan Faulkner, David Raposo, Jack W. Rae. Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy P. Lillicrap. Relational recurrent neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7310-7321, 2018
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(1 ((11))))67-668)) 1997.
Murray Shanahan, Kyriacos Nikiforou, Antonia Creswell, Christos Kaplanis, David G. T. Barrett, and Marta Garnelo. An explicitly relational neural network architecture. Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 1 119 of Proceedings of Machine Learning Research, pp. 8593-8600 8603. PMLR, 2020.
David Silver, Thomas Hubert, Julian Schrittwieser, Toannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017.
Thiago Simão, Nils Jansen, and Matthijs TJ Spaan. Alwayssafe: Reinforcement learning without safety constraint violations during training. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1226-1235, 2021.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Pashootan Vaezipoor, Andrew Li, Rodrigo Toro Icarte, and Sheila McIlraith. Ltl2action: Generalizing Itl instructions for multi-task rl. International Conference of Machine Learning, 2021.
Min Wen and Ufuk Topcu. Probably approximately correct learning in stochastic games with temporal logic specifications. In IJCAI, pp. 3630-3636. 2016.
Zhe Xu, Ivan Gavran, Yousef Ahmad, Rupak Majumdar, Daniel Neider, Ufuk Topeu, and Bo Wu. Joint inference of reward machines and policies for reinforcement learning. In Proceedings of the International Conference Automated Planning and Scheduling, volume 30, pp. 990.598.00 2020.
12

---

Published as a conference paper at ICLR 2022
Haonan Yu, Haichao Zhang, and Wei Xu. Interactive grounded language acquisition and general- ization a world. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC. Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap. Edward Lockhart, al. Deep reinforcement learning with relational inductive biases. In International Conference on Learning Representations, 2018.
13

---

Published as a conference paper at ICLR 2022
Algorithm Symbolic Module (SM)
1: Input: Instruction T
2: Generate the accepted sequences of tasks K +- E(T)
3: Retrieve the current observation:
4: Get the true proposition: p + L1(2)
5: Get the first task:
P(K,p)
6: repeat
7: Get the next action: Neural Module (NM))2,a)
8: Execute which updates
9: Get the new true proposition: p +- Cr(z) Provide the reward: NM + R(pp)
10:
if p = Pa then
12:
Update K,, < p)
end if
14:
until K. -==- !
A THE SYMBOLIC MODULE
In this section we include the pseudo-code of the SM and the internal extractor (E) and progression (P) functions, whose procedures are explained Sec. 3.2 of the main text.
Algorithm/ shows the general functioning of the SM and its iterations with the NM. Within the SM, Algorithm details the Extractor E. Given the complex instruction T, & generates a list K. with all the possible sequences of tasks that satisfy T. Note that are the tasks that the neural module (NM) expected to learn/solve compositionally. Algorithm refers to the Progression function P, which updates K. according to the true evaluation Pa given by the internal labelling function L,, that indicates to the SM how the previous task has been solved, and selects the next task a'.
B ENVIRONMENT DETAILS
Below expand the description of the benchmarks introduced in the main text:
Minecrafttinpired. In this environment agents have four actions: Up, Down, Left, Right to move one square from their current position, according to the given direction. Specifically, we use a configuration where maps are procedurally generated by randomly selecting populations of objects, their placement and agent's 's starting position, and where each tile within has a resolution of 9x9 gray channel) pixels. We generate 50 objects split into various sets. Each object is represented by matrix of 9x9 values. These values were procedurally generated with pseudo-aleatory numbers (we used fixed seed). The global
objects referred as X, where the total number of objects is X = 55. The set is partitioned into pretraining set [Xal = 35 training set |2& = 20 and a test set Xg = 20, where X' X, X3. X² C X₁ and X; O X33 = 0.
MiniGrid. In this environment there are three possible actions: Move Straight, Turn Left, Turn Right. Here objects are formed from compositions colors and shapes. We expand the number of shapes and colors from vanilla MiniGrid to have eleven colors and eight shapes. We refer to the total set of colors and shapes as C and F respectively. We also divide the global set of instructions in four categories follows:
Reachability atomic goals, which have the form "True U + p".
Negative reachability goals, e.g., "True U - P1 V -p2" or "True - pp".
Positive instructions such as "+ P1 + +p2" or " P1 V +p2 + Ps V +p1".
Negative conditions, e.g.. P1 U P2 +ps".
With atomic reachability goals
sets C1 and F₁ while training and C2 and Fy for testing, where
|C = 8, [F 6, C' = C, U C2. C C = O. F = F, U F2 and F, n F2 = 0. The remaining
categories of instructions C3. Fg in training and Ca. F4 for testing, where |Ca = 8. [Fl =
14

---

Published as a conference paper at ICLR 2022
Algorithm 2 Extractor function, obtains the list of all the possible sequences of tasks
1: Function &, Input: T
2: Initialize the list for the sequences of tasks K
3: for each atomic task E 7 do
4: if a is atomic positive or negative then
5: for all Seg K do
6:
Seq.append(a)
7: end for
8: else
9:
// There are non-deterministic choices
10:
Initialize choice list: CL
LK t (eC) for all Seg K do
12:
for all choice E a do CT..apend(choice)) Generate a clone per choice Seq' t Seq K.append (Seg') end for end for
14:
15:
16:
17:
18:
19:
Initialize counter c < -1 for in range(leng(h(K)) do
20:
21:
5% LK = then c+=1
23: end if
We append different choice to each sequence cloned apppaa (CL[cc)
25: end for
26: end if
27:
end for
28:
return
C= C=GUCaCnna=-.FF and F4 C Fi. Intuitively, we use reachability goals to ground some shapes and color for our agents, and the rest the instructions for the remaining elements. Then we test the agents with the unseen colors and shapes for each category.
and F Also, C2 c C3, C,
Task generation. We procedurally generate tasks detailed in Algorithm 4. Where task type (Atepe) refers to one of the four categories of instructions as detailed in MiniGrid paragraph in Appendix B. Note that in Minecraft all task types have the same population of training objects since all the test objects are zero-shot.
Experiment Architecture. All the agents in this work use a pretrained encoder. While pretraining use only instructions of the form ol trueUl, with objects from X, in Minecraft, or Ci and Fi in MiniGrid. This pretraining stage helps the encoder to differentiate objects within the environment. Once pretrained, the encoder remains frozen to prevent overfitting. The training stage is 70M timesteps in Minecraft or 50M in MiniGrid, with maps of training size and populated with objects from the training sets. For testing, all the training parameters are frozen and the agents are evaluated in sets of 500 maps populated with OOD objects from the test sets. At the beginning of each episode, a task a is procedurally generated (e.g., 0i or a2 above). Train and test tasks are generated with objects from the corresponding set. In Minecraft, tasks include zero-shot (i.e., unseen) objects. Once we selected a task, a new map is generated where the agent is placed in an aleatory position, some all the goal objects are randomly placed and some or all the objects for the constraint are also randomly placed. Additional objects from the corresponding set are also included in the map as distractions.
15

---

Published as a conference paper at ICLR 2022
Algorithm 3 Progression function, returns the next task to be solved.
1: Function P, Input:K,, Pa.
2: if Po + then
3: for all Seg E KC do
4: Pa fulfills Seq [0 then
5:
Seq.pop(Seg(0)
6: else
7: K.pop(Seq)
8: end if
9: end for
10: end if
a'
12: if I = then
return a' 14: else
15: Select the head of the first sequence: a' + [c[
16: for all Seg K. do
17: // look for non-deterministic choice
18: if a' & Seq(0] can be combined then
19:
a' - a' U Seq
20: end if
21: end for
end if
23:
return a'
Algorithm 4 Task generator
1: Input: is_test
2: Randomly select task type Ottype
3: if is test then
4: Pop + Collection of objects for Ottype
5: else
6: Pop +- Collection of train objects for Otype
7: end if
8: Randomly select number of goals ng in range [1, 2]
9: if Oktype ! 1 then
10: Defaults number of safety constrains: n. =
11: else
12:
Defaults number of safety constrains: D.
13: end if
14: if Otyppe ==3 3 then
Randomly select ne in range [1,
16: end if
Form an instruction / with ng goals and n. safety constrains by selecting different objects from Pop for every constrain and goal.
18: return
C EMPIRICAL DETAIL
Here include further details about the empirical evaluation. The repository at https: / //github http:///we.eooeaeeeeeeoooooaaaae provides the code used to per- form experiments.
C.I HARDWARE
For training we use various computing clusters with GPUs such as Nvidia Tesla K80, and GTX 1080. We employ Intel Xeon CPUs and 7700k and consume 5 GB of RAM per every three independent
16

---

Published as a conference paper at ICLR 2022
100
MLP RN
NHA
MLP
PrediNet
00
PredNel
RN
Random Walker
MHA
Training Plooart
Training Rowward
41 WHf WHHassppee
80
80
40
%
10M
20M
30M
4DM
S0M
EDM
70M
%
20M Numbor of training slops
20M
401
SOM
Numbor of training stops
100
100
MLP
MHA
PrediNet
80
RN
PrediNet
RNFG
Random Walker
MHHA4
Roward
60
0
10M
4DM
%
30M
Number ttakiing
100
100
BRIM BRIM
ResBRIM
BRIM BRIM
ResBRIM Random Walker
Random Walker
Training Reward
Traiiig Roward
20
10M
2014
3014
40M
50M
S0M
70M
201
S0M
SOM
Mumber of training stops
Number of training sleps
Figure 7: Training plots of the networks studied in Sec 5. Continuous lines correspond to the 50th percentiles while the shadowed areas are the t5th and 75th percentiles. Note that good train performances do not yield to similarly good test results as detailed in the main text. Left: Minecraft training plots. Right: MiniGrid plots. Note that general performance drop at 20M in this setting is motivated by the higher chance of generating larger training maps from that point on.
runs working in parallel. Running concurrently, each experiment typically takes 3 days to train, days in the case of latent-goal configurations.
C.2 RL IYPERPARAMETERS
All of our agents trained with A2C. Specifically, weight of 0.5 and batch size of 512. We work with a scheduled learning rate using a starting value of 85-3, -5 at the step 30 M. and 4e 55 M. In the case of BRIMs, which larger networks, we use a constant learning rate of 3e- These values offered the best results when testing after gric-search
discount factor y 0.99, value loss
significantly
range values between le and 1e - The entropy eeplopy-explloaatin term
was fixed to H = le after testing with values in the range [le 0, 5,
C.3 NETWORKS
Our models use 3-layer convolutional networks in both settings. Particularly, Minecraft we use kernel of 3x3 in the first and second layers and 1x1 in the third one. The number of channels in
17

---

Published as a conference paper at ICLR 2022
((aa)
top layer
residual connection
hig
Task
Central module
Encoder
Obs
BRIM bottom layer
ar-1
(az)
Central module 2
L
BRIM top layer
Ve
Encoded
h
ho Lig
Task
Obs
Encoder
Central module 1
Obs
Encoded
BRIM bottom layer
Task
Figure 8: Top: vanilla BRIM and ResBRIM (without and with the residual connection respectively). Green cells within the BRIM layer refer to activated modules while gray cells are deactivated. The bottom layer receives imput the output of the central module and the hidden state from the upper layer in the previous time step (h/_1). The upper layer receives the output from the bottom layer (h+). Only strong connections (the ones between activated modules) are shown. In the case of ResBRIM the upper layer also receives the output of the Central module. Bottom: BRIMLG The architecture similar to the ResBRIM but the top layer receives as additional input an embedding from the observation exclusively (L.). Consequently, the top layer has only access to the current goal through the lower layers of the hierarchy.
each layer are 8, 16 and while strides are 3, 3 and respectively. Note that this encoder has been designed bearing in mind the resolution of the tiles within so that the latent representations (i.e., the outputs of the encoder) of the tasks are independent from the latent representations of the original observation This inductive bias is exploited in the latent-goal architectures. In MiniGrid all kernels and strides are of size 2. The number of channels are 16, 32 and 32 respectively. Note that encoders are pretrained and then frozen, we found that doing this improved the performance of all agents. With MiniGrid's 's text instructions we use bidirectional recurrent layer of size 32. In multi-layer configurations the other two recurrent layers are LSTMS of size 128. The information bottleneck is FC layer of size 16.
For the central modules the multilayered architectures, we use the best hyperparameters from previous ablation study in Shanahan et al. (2020). Particularly, in the case of the MLP we use single fully-connected layer with 640 units. The RN has a central hidden size 256 units and an output size of 640 units. The MHA employs 32 head with key size of 16 and value size of 20. Last, the PrediNet has 32 heads, with key and relation sizes 16. The latent-goal variants use a FC layer of size 16 for the information bottleneck. Regarding BRIMs we use the hyperparameters recommended for RL in Mittal et al. (2020). Specifically, we use 2 hierarchical BRIM layers, with 6 modules of 50 LSTM units each. Four modules (k) are active illustrates the different BRIM configurations that we evaluate Sec. in the main text when working with visual instructions, i.e., the Minecraft setting.
given time step (top-k mechanism). Figure 8
18

---

Published as a conference paper at ICLR 2022
MiniGrid- Train
Minecraft-Tril Train
300
Map aime
Mag
7X7
7631
2702
RN
[ours]
MLP
MLP Yjurs)
(our)) IN (HNA
(Mur (rudiltt"ouns]
MHA
[ouns
Prediket
(rod
MHA
ProdiNet
Figure 9: Results with training instructions in 500 maps (per size) of different dimensions. Sizes 14 and are OOD.
MiniGrid-Train
Minecraft-Tran
80
100
Normalized Rewards
lized Rewards
80
40
Normalized
40
20
Map size 7x7
14x14
22x22
Map size
7x7
14x14
22x22
Random
BRIM
ResBRIM
(ours
Random
ResBRIM
(RIM
Figure 10: Results of i.r. of vanilla BRIM (BRIM), BRIM with residual connections (ResBRIM), and with LGs (G)IMLG).
D ADDITIONAL EMPIRICAL ANALYSIS
In this section we first include the training results and tables with the specific values used to generate the bar-charts. Then, detail the results of the statistical analysis two-way ANOVA) of the results with multi-layered networks. Last, we provide the training plots of the experiments in the main document.
D. TRAINING PERFORMANCE AND DETAILED VALUES
Figs. 9-11 illustrate the training results of the experiments in the main section. Tables 1-4 provide the specifics values in a table format. From the training results, we observe that latent-goal configurations
Ablations train maps: Battleneck
Ablations train maps: Lg
100
90
Normllted Rewards
50
Map size
14x14
22x22
22x22
Not- Not-Task-Agneet
Task-Agnostic
Figure 11: Ablation studies with PrediNe in Minecraft (5 i.r. per variant). Left: Impact different bottleneck sizes. Right: Impact of having L, task-agnostic not.
19

---

Published as a conference paper at ICLR 2022
Table Results with either training (train) OOD instructions (test) in 500 maps (per size) of differ- ent dimensions. Note that sizes 14 and 22 are OOD for all the agents. MC refers Minecraft whereas MG to MiniGrid. Results show the average reward and standard deviation from 10 independent runs (i.r.). Values are normalized so that 100 refers to the highest performance achieved by the best run globally in maps of the given size and benchmark. Best average rewards each setting and size bolded.
Mup-size
MLP
RN
MHA
ProdiNet
Train
Test
Train
Test
Train
Test
Train
Test
MG-7x7
18.2(5.7)
14.1(3.6)
20.6(5.4)
18.7 (6.4)
45.9(26.6)
9.2(3.7)
19.1(12.2)
15.9 (7.3)
MC-7x7
33.4(7.9)
21.1(3.2)
5(1.9)
(1.6)
41.1(7.7)
36.6(4.3)
21.8(3.7)
(6.1
MG- 14x14
28.6(8.2)
21.1(5.7)
28.6(6.2)
27.5(6.0)
57.3(17.8)
21.9 (4.7)
31.2(10.7)
25.6(7.3)
MC-14x14
38.7(6.5)
32.0(4.5)
8.3(2)
7.9(2.4)
51.7(9.3)
45.8 (7.6)
40.2(5)
31.9(6.2)
MG-22x22
27.8(8.6)
21.6(4.3)
26.9(10.1)
24.3(5.1)
54.9(23.6)
22.7(6.8)
32.1(8.9)
(4.1(8.0)
MC-22x22
47.8(3.6)
37.8(3.8)
9.2(1.5)
8.3(1.8)
56.9(10.7)
49.2(5)
48.7(10.3)
38(8.5)
Map-size
MLP40 (ours)
RN (ours)
MHA (ours)
PrediNet (ours)
Train
Test
Train
Test
Train
Test
Train
Test
MG-7x7
23.6(5.5)
21.6(2.6)
22.6(3.3)
21.8 (2.8)
42.0(24.9)
15.4(3.5)
32.1(9.3)
(6.1(5.8)
MC-7x7
40.2(5.7)
24.7(2.6)
4.4(1)
4.2 (0.7)
35.2(5)
31.6(5.3)
56.2(17.6)
29.5(8.2)
MG-14x14
31.((1.)
24.6(3.8)
30.1(6.1)
29.2(6.9)
50.4(14.9)
(9.4)
37.0(6.7)
30.6(5.5)
MC-14x14
41.7(5.1)
35.((4.)
13.1(2.0)
11.5(2.2)
48.1(4.4)
41.8 (4.9)
67(5.2)
51.7(6.5)
MG-22x22
31.((63)
22.5(5.0)
34.2(2.3)
25.1(1.5)
53.3(21.4)
25.4(8.2)
39.9(2.7)
29.7(7.0)
MC-22x22
52.6(8.6)
41.(7777)
117(1.5)
10.7(2.1)
58.4 (1.5)
51.5(5.5)
77.5(9.1)
65.3(6.1)
Table 2: Results from 5 i.r. of a vanilla BRIM (BRIM), BRIM with residual connection (ResBRIM) and BRIM with latent goals (BRIML9). Best results for each setting and size are in bold.
Map-size
BRIM
ResBRIM
BRIM (ours)
Random walker
Train
Test
Train
Test
Train
Test
MG-7x7
19.6(6.2)
(4.1(4.7)
19.2(4.8)
16.7 (3.6)
23.8(5.4)
17.3
4.3(0.4)
MC-7x7
27.9(6.3)
15.6(2.8)
58.6(10.4)
15.8 (2.3)
77.2(19.5)
26.4(7.0)
3.7(0.1)
MG-14x14
25.3(5.4)
19.9(6.2)
33.7(3.9)
(1.4)
38.8(9.8)
26.6(5.6)
9.5(1.3)
MC-14x14
37.9(8.2)
26.5(6.7)
63.5(5.5)
39.6(7.0)
77.3(21.3)
46.8 (8.1)
11.5(1.))
MG-22x22
26.5(7.1)
20.0 (7.6)
27.9 (3.0)
25.3(2.1)
48.5(26.0)
30.3(6.6)
13.6(4.2)
MC-22x22
41.5(8.4)
29.3(4.1)
60.6(10.4)
(8.7()
79.8(17.0)
58.4(14.8)
11.1(0.5)
can improve the performance also the with training instructions and that this improvement becomes more noticeable as environments become larger in size, i.c., as we get further from the training distribution of environments. The fact that the effects of latent-goal networks are most noticeable with unseen instructions aligns with the conclusions drawn from the main document, where we stated that this architectures are specially conceived to strength agents' ability to generalize to new tasks.
D.2 STATISTICAL ANALYSIS
To confirm that the standard error of the different conditions do not overlap each other, we perform two-way analysis of variance (two-way ANOVA) with the independent variables being the neural networks used the central modules and the architecture configuration, i.e., latent goal (ours) or standard configuration. The 2-way ANOVA is done across the maps of different size of Minecraft and MiniGrid respectively. Note that evaluate the different hypotheses, we need to check whether the F value is greater than its corresponding F-critic or not, and to confirm that the P-value is smaller than Alpha.
Table 5 shows the results of the ANOVA analysis with the results from Minecraft. We see that it confirms that both the architecture configuration and the neural network have a significant impact of the performance of the agent. Noticeably, the ANOVA results show that there is not significant interaction between the two variables, meaning that the neural network is not relevant in the general improvement in performance that latent goals grant in this setting. Still, note that this is analysis of the statistical impact averaged across all the networks. As studied in Sec. 5.1 the latent goals do have different impact with the PrediNet than with other networks.
20

---

Published as a conference paper at ICLR 2022
Table 3: Ablation studies in Minecraft (5 irr. per variant). PrediNettosub and PrediNethoPos are variants of PrediNethG without the element-wise subtraction and the feature coordinates respectively. PNMHА uses an MHA in CM2 while MHAPNIC uses an MHA in CM1. Best results are in bold
Ahlations
7x7
14x14
22x22
PrediNet variants
Train
Test
Train
Test
Train
Test
PrediNettasa
41 (8)
22.8 (2.8)
64.7 (7.9)
42.3 (6.9)
63.7 (7.8)
46.4 (8.1)
PrediNethsSe.
50.6 (9.9)
29.3 (5)
68.2
(5.4)
70.6 (9.3)
56.4 (9.2)
PNMHA
(7.6)
(6.2)
52.8 (3.9)
46.5 (7.9)
(6.6)
45.2 (10)
MHAPNI6
(10
(7.5)
59.9
(4.9)
(6.4
51.6 (7.6)
Table 4: Ablation studies with PrediNet in Minecraft (5 i.r. per variant). In the first set we explore the impact of different bottleneck sizes. In the second we contrast having Ly task-agnostic not. For every ablation set, best results are bolded.
Ablations
7x7
14x14
22x22
Bottleneck size
Train
Test
Train
Test
Train
Test
4
57.0 (8.4)
26.0 (10.9)
72.0 (15.7)
60.4 (5.9)
74.8 (17.2)
58.3 (12.9)
16
51.2 (11.6)
25.3 (6.1)
81.7 (17.9)
49.4 (14.2)
86.1 (12.6)
(10.))
64
49.8 (10.2)
19.1 (5.7)
69.3 (14.3)
35.2 (13.9)
64.2 (15.1)
38.1 (13.3)
256
56.7 (14.7)
(11.))
70.5 (13.0)
(3.5)
71.7 (10.1)
(17.3
La data type
Task-agnostic
51.2 (11.6)
25.3 (5.4)
81.7 (17.9)
49.4 (12.6)
86.1 (12.6)
61.8 (10.3)
Not Task-agnostic
48.1 (13.6)
23.0 (4.6)
58.7 (10.4)
33.7 (5.8)
59.4 (14.1)
44.4 (7.3)
Table shows the results of the ANOVA test with MiniGrid. Here we see that neural networks have significant impact in the training performance of the agent but that is not the case with the architecture, i.e.. using latent goals does not have significant impact in the global training performance. Nevertheless, we see that this changes with unseen instructions (test) where the latent goals have stronger impact performance than neural network, being both statistically relevant. Last, from the interactions' results variables have a significant impact in the final performance, i.c., the effect in performance that latent goals have is strongly dependant the neural network and vice versa.
that both in training and test the interaction of the two
Regarding the differences in how network and architecture impact in Minecraft and MiniGrid, we believe that these are motivated by the additional difficulty of MiniGrid observation and action settings. Specifically, MiniGrid only allows agents to move forward or turn around, and agents can only observe what is in front of them. This requires agents rely more their memory than they do in Minecraft, where they can observe the objects around them in any direction and have actions to move in four different directions. This point also motivated the need of using an schedule in the introduction of larger training maps in MiniGrid, as detailed in Appendix D.3
D.3 TRAINING PLOTS
Figure 7 shows the training plots of the standard (top), latent-goal (middle) and BRIM (bottom) configurations in Minecraft (left) and MiniGrid (right). In the later can be appreciated a general decrease of performance the 20M mark. This is because we noticed that in MiniGrid, where the navigational features force the agent to further rely on memory, we noticed that agents struggled when learning with the default configuration of random sizes with n [7 10). To alleviate this, through the first 20 M steps we give a higher chance (70%) to maps of size 7, to help agents to learn from the instructions before navigating through larger maps. Note that good training performance, as shown in these training plots, does not necessarily mean good OOD performance. This is highlighted in Table 1.
21

---

Published as a conference paper at ICLR 2022
Table Two-way ANOVA test with Alpha= 0.05 in Minecraft, Neural network refers to the impact of the different neural networks in the central modules, whereas architecture configuration refers to the impact of whether using a latent goal or a standard configuration.
Train
F
P value
F critic
Architecure configuration
43.56
2.74exp 10
3.88
Neural Network
237.31
2. 2.13exp- 70
2.64
Interactions
31.27
5. 5.16cxp 17
2.64
Test
F
P - value
F critic
Architecture configuration
23.63
2.1xexp -6
3.88
Neural Network
166.75
1.26exp 557
2.64
Interactions
16.26
1.26exp 09
2.64
Table 6: Two-way ANOVA test with Alpha = 0.05 in MiniGrid.
Train
F
P- value
F critic
Architecture configuration
1.37
0.24
3.93
Neural Network
39.57
1.63xp-111
2.69
Interactions
1.86
0.14
2.69
Test
F
P value
F -critic
Architecture configuration
8.57
0.004
3.93
Neural Network
5.51
0.001
2.69
Interactions
0.53
0.6
2.69
E COMPOSITIONAL LEARNING
Here we assess whether agents are learning compositionally from SATTL instructions and check they are really following the given safety constraints when applied to OOD objects or if they are just learning reach the goal Ja- We focus our study on their performance with negated constraints, since previous research highlighted as the most challenging type of instructions Hill et al. (2020); León al. (2020). Compositional learning Lake (2019) (also known as systematic learning Hill et al. (2020)) refers to the ability of understanding and producing novel utterances by combining already known primitives Chomsky & Lightfoot (2002). In our context, an agent that learning compositionally should be able to solve the instruction -c U + if the primitives e and are known and it already learnt to solve -LU + +p'.
Table 7 shows the results from a control experiment where we track the performance of the agents according to tasks of the form -cU + p when giving reliable, partially occluded or deceptive instructions. Specifically, every row shows the test performance of the multi-layered networks trained in Sec. 5 when providing rewards according to the instruction -CU + p, which intuitively means "avoid until reaching p". The first row shows the performance of the agents when the SM provides reliable instructions to the NM, i.c., the instruction provided is the one used to generate rewards. The second row shows the performance under the same setting, but where the extended observation provided to the NM is a reachability goal (true U + p). Intuitively, for the second row the agents are given the right goal p but not the safety constraint, i.e., partially occluding information of the real task. The third row shows the performance when the SM gives deceptive information abut the safety constrain (+cU + p), i.e., the SM is "telling" the NM go through the objects that actually should be avoided. Agents learning systematically should show worse performance with partially occluded instructions than with the reliable ones, but still better than a random walker. Additionally, the worst performance should come when provided deceptive instructions.
From Table 7 we see that all the variants using the MHA and MLP modules follow the rule 1s row > 2nd row > random > 3rd (compositional rule, or c.r. for short). Notably, the best performance comes from a variant that does not use relational networks nor attention (MLP*.)) Still, this does not imply that MLPs are better suited for negation since agents were trained in much wider variety of tasks and the MHA, MHA LG and ! outperformed the MLP in the general evaluation test. In the case of the RN, the c.r. is satisfied but the general low performance and the small difference between the results of the first and the second rows (14 and 13.3 respectively) indicates weak generalization. This is worse with the M-RN and the PrediNet having both equal better
22

---

The syntax of LTL,, is defined as Published as a conference paper at ICLR 2022
Table Study on compositional learning with zero-shot objects and instructions. Results show the performance obtained by each network in 200 test maps when rewards are given according to the "real goal", while the symbolic module provides the "given instruction" agent learning compositionally should have 1st row > 2nd row >> 3r row for the values within its column. Also, values lower than random are only acceptable in the third row (deceptive instruction). The mean performance of random walker is 12.5 independently of the "given instruction".
the neural module.
Real goal: - +p
instruction
MI.po
RNUG
MHA
MHA
PrediNet
PrediNet
44.6(10.5)
70.6(11.9)
14(1.1)
(1)
55.8(6.1)
43.8(7.3)
17.5(1.2)
43.6(9.3)
U +p
20.345.7)
18.064.6)
13.3(0.9)
13.9144)
16,9(30))
(6.3)
19.9(34)
17.5(38)
tc +
8.7(1.6)
11.0(2.6)
7.9(2.2)
5.6(1.)
10.8(2.6)
15.0(3.6
8.2(1.8)
8.4(1.2)
performance with partially occluded instructions than when receiving the real task as input. Such results imply that these networks have correctly learnt generalize negated instructions. This is not the case with the PrediNerLG whose results correctly follow the c.r. In addition, the worse performance of the PrediNet than the MHA with partially occluded and deceptive instructions suggests that the PrediNethG advantage over the MHA in maps of OOD size ( see Table comes from a better ability of the former to execute safety constraints when compared with the later. Note that the larger the map the harder it is to find thus the bigger chances of accumulating penalizations due to "violations" of the safety condition.
Discussion To the best of our knowledge, the only two works that include some empirical evidence of emergent compositional learning with negation, i.e., achieving performance 50% better than chance are Hill et al. (2020); León al. (2020). However, in those works negated instructions are interpreted as "something different from", e.g., "not p" intuitively meant "get something different from p". Instead, here have interpretation of negated atoms more aligned with propositional and temporal logic, and also to natural language, where "not p" intuitively means that p must be false. Hence, we believe this is the first work to show that DRL agents capable of learning the abstract operator of negation in its classical interpretation and successfully apply it to new utterances.
F SATTL AND LTL,
We stated that SATTL is fragment of the widely-used LTL,,. follows:
: p pl 10 A 42 Osp 1 Up2
Since both LTL, and SATTL are defined ofer finite traces, we can directly introduce the following translations:
6 ::= p p1 41 N 42
O4p
Definition 3. Translations from Task Temporal Logic to L.Tl., are defined as follows:
T(a)
wr
(TUUT)
T(T) v T(T')
(WW A AT(T") (T2:TT)
ifT a ifT = Ti: iTT = Ti UT2
T(T;T")
((TT;;""
We immediately prove that translation 7 preserve the interpretation of formulae SATTL. Proposition 1. Given model N and trace X, for every formula T in SATTL, = iff (N,)) += r(T)
Proof. The proof is by induction on the structure of formula T. The base case follows immediately by the semantics of SATTL and L.T.y.
23

---

Published as a conference paper at ICLR 2022
As for the induction step, the case of interest is for formulae of type T; T". In particular, (N, F
T; iff for some و < |A,, (W, X[0, j]) + T and (N, + (J) F TV By induction
hypothesis, this is equivalent to (W A(O, il) -= WT and (N, + L, (A]) F T(T) in the case of -UT' Finally, this is equivalent to (N) = WT (T")). The cases for = =T; 72 and T= Ti U 72 are dealt with similarly.
Finally, the case for T TU follows by induction hypothesis and the distributivity of V.
24


Reviews:
[{"review_id": "uEQhPvICdCA", "reviewer": "Reviewer_Ymus", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Major Comments**\n\n*Sound policy network design choices:* The design choices in the proposed latent goal representations appear to be sound. It appears that joint training of the goal specific information with the task embedding is beneficial to generalization. And this design choice appears to be benefiting the performance irrespective of the network architecture of the central module. However, this is hard to judge considering the presentation of the results.\n\n*Training and testing on multiple formulas:* I like the fact that the authors have sampled multiple instructions to test on and not on a few handpicked formulas. The authors should provide the definition of the sampling algorithm for the instructions in the appendix.\n\n*Presentation of the results:* My biggest issue with the paper is how the results are presented and analyzed. I urge the authors to use bar charts instead of raw numbers to present the results as the comparisons are much easier for the readers. Further I believe that the standard errors of many of the conditions overlap with each other. The authors should perform statistical analysis appropriately before claiming superiority of any of the conditions over the others. With so many experiments, and data points, sound statistical analysis is important. a 2-way ANOVA (independent variables being network architecture, and latent goal representation or the classical architecture) with repeated measures over the minigrid and minecraft domain seem to be the appropriate statistical approach. Without this, the claims of benefits of one over the other are tenuous. While the authors can include this in the final version, I believe the authors should be able to run this before the responses are due.\n\n**Minor comments:**\nPlease consider the following suggestions for clarity:\n1. I would suggest the authors to bulletize the baseline architectures, instead of writing a long paragraph. That would make it easier to parse the important features. \n2. Separate discussion of the results from the ablation studies, right now you only get to the ablations at the end of that subsection\n3. I would have liked to see some contrastive description of SATTL with respect to TTL. How does SATTL improve the expressivity of TTL? What types of task cannot be expressed in TTL but can be in SATTL?\n\n\n", "summary_of_the_paper": "This paper lies in the space of using non-Markov reward functions to model temporal task, and in addition to that attempts to generalize instruction following beyond single instruction provided in a formal language. The authors propose a dual network solution to embed the environment representation and separately a goal specific reasoner grounded within the environment. For planning with temporal formulas, the authors extend prior work on Task temporal logic. Although the algorithm for this remains largely unchanged from prior work (please clarify if this is wrong).\n\nThe results demonstrate that the latent goal representation helps in generalizing at both the object level and at the task level. However the presentation of the results can be significantly improved (please refer to the main review).", "summary_of_the_review": "I believe that as written the paper is not ready for publication. I believe that statistical analysis of the results is necessary, and can be reasonably completed in the response period. I would be willing to upgrade my scores (correctness and significance) based on that. I also believe that the edits for clarity would significantly strengthen the paper.\n\nUpdate: The authors incorporated most of the comments, and the statistical analysis bears out the major claims.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "rKwyluiRDdP", "reviewer": "Reviewer_FvKn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Section 3.2: This approach, especially with the definition of the symbolic module, seems to bypass many of the challenging problems in instruction-following: sparse rewards (e.g., rewards only available at the end of the episode), with no explicit progress-monitoring provided, noisy instructions and environment, no basis for task-specific reward-shaping, goals are not necessarily conditioned on the agent’s current state (i.e., leading to issues with initial heading, localisation, etc.). The SM provides a lot of privileged information, through its labeling function and comprehensive reward function.\n\nSection 3.2: How is ambiguity handled — i.e., when there exists multiple ways that T can be entailed?\n\nSection 3.3: The manuscript states \"Consequently, it can be beneficial for agents to compute both human instructions and sensory inputs in a dedicated channel, generating a latent representation of the current goal. We call this type of deep learning model a latent-goal architecture.\" By itself, this is not novel: there are a plethora of works in learning robot skills, and they have dramatically less supervision.\n\nSection 4: Table 1: Train/Test results should be compared separately. Bolding schemes (to highlight the best-performing model) change, from table to table; this should be made consistent.\n\nSection 4: The manuscript, which pursues so-called 'latent goal architectures' for following formal instructions, somehow makes reference to neither the robot skills literature nor the robot instruction-following literature. Additional experiments and comparisons are needed.\n\nThroughout: The manuscript relies on the assumption that \"human instructions can be separated from the sensory input\". This is a strong and unrealistic assumption, in practice. What need to we have, then, for multiple sensor modalities, when the instruction (whether natural or formal language) can completely specify the task?", "summary_of_the_paper": "The manuscript proposes 'latent goal architectures' for following formal instructions in OOD scenarios.", "summary_of_the_review": "I have some issues with the novelty of the work presented, the accuracy of the claims made, and the somewhat lacking comparison with the related work and key ablation experiments. See main review, above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "oxifOWw0abn", "reviewer": "Reviewer_bGYm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In my opinion, this paper is well laid out and tackles the important problem of creating agents that can perform high level temporal tasks. While a fairly incremental contribution, the thorough empirical analysis indicates that performing out of distribution temporal tasks is possible.\n\nOne thing I was surprised was not explicitly done was to use the underlying automata structure of proposed reward function to further accelerate the RL loop. In particular, I recent work on reward machines (which can encode regular languages) uses a trick similar to hindsight experience replay to provide counterfactual examples that and mitigate reward sparsity issues. Looking at the training plots in the appendix, I wonder if the generalization may have occurred if many fewer steps. I am also curious if such a trick could be used to remove the small (and seemingly adhoc) penalization incurred at each time step (which is in theory redundant with the discount rate, but a source of possible reward bugs).", "summary_of_the_paper": "This work focuses on training (via deep RL) task conditioned agents for a given class of workspaces. Tasks are formalized as a variant of finite temporal logic (and thus encode a regular language). Tasks are either provided as text or via visual diagrams. A key goal of this work is to perform well on tasks that lie outside of the training distribution. The paper illustrates on two common grid like domains with the results of a number of various architectures and ablations reported.\n\nThe approach taken is an adaptation of deep reinforcement learning, with the primary contribution being a slight, but seemingly important, modification of an existing architecture to create a separate embedding for (i) a task agnostic state encoding and (ii) create a task specific state encoding. Furthermore, in order to encourage the network to generalize between tasks, an information bottleneck is placed on the task specific encoding.", "summary_of_the_review": "Overall, while somewhat incremental, I think this paper adds a valuable contribution, is very thorough, and pretty coherent. The focus on simply modifying the inductive biases in the architecture (rather than dramatically changing training) is also a good venue fit.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EPP6bmUA-Fk", "reviewer": "Reviewer_2S8A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nIt seems like a good idea to include environmental information in the reward specification phase, and empirical results do show that this is effective.\n\nWeaknesses:\n\nAt present the paper suffers from severe clarity issues, which make it difficult to evaluate. Specifically:\n\n- The abstract doesn't actually explain the task or method, and reads more as a collection of jargon. This needs to be reworked to more clearly specify the problem formulation, goal and proposed solution.\n\n- Contributions need to be made clearer\n\n- Problem formulation: the paper is in need of a clear problem formulation early on, explain exactly what the task is, inputs and outputs, and key steps in the solution, early on. Right now, the background provides an MDP definition that could be repurposed to explain the task, and motivate the key reasons for the proposed framework much earlier on.\n\n- SATTL section: it's unclear what this adds to the paper - right now this reads as a floating intro to SATTL that just breaks the flow of the text. I'd move this to an appendix, or rework to explain only what is required from the perspective of reward specification.\n\n- I only started to understand the paper in section 3.3 - I'd consider moving this much earlier, and motivating the problem using this example.\n\n- It's difficult to map the framework in 3.2 onto the architecture in Fig 1. The method description needs a lot of work.\n-Fig 1 and the associated discussion is very confusing. Right now, I'm unsure if the proposed solution only operates in the reward specification phase, and is then used to optimise an actor-critic policy, or if something interesting happens in the latent state of an actor critic network, or if a meta-agent is being trained to predict a zero-shot value function and policy. \n\nOverall, I believe this paper would benefit greatly from being re-read from the perspective of someone not involved in the project, because at present it does not adequately communicate the work completed.", "summary_of_the_paper": "This paper introduces an neural architecture for a DRL agent tasked with learning to solve temporal logic specifications. Results show the proposed architecture outperforms existing ones when it comes to performance on instruction following in unseen environments.The proposed architecture takes in an observation and a task description, which appears to be transformed into a reward function that is then optimised. The key contribution appears to be around the inclusion of some latent state information about the environment in the reward specification phase, with the motivation that this makes it easier to interpret a human temporal logic specification of a task.\n\nResults appear to be strong, and do show that this results in substantial performance increases in unseen tasks.\n\nUnfortunately though, I found this paper a difficult read and in need of substantial work before it is ready for publication, primarily around the problem framing, method description, discussion and motivation. At present, it lacks the clarity required to best showcase the results, and I cannot confidently say I understand this paper, because it is not explained or presented clearly enough.", "summary_of_the_review": "I think there is interesting work in here, and a nice story to be told. Unfortunately, the current structure of the paper is not good enough to support this, and the paper is in need of a substantial amount of rework to improve it's clarity.\n\nAt present, it lacks the clarity required to best showcase the results, and I cannot confidently say I understand this paper, because it is not explained or presented clearly enough.\n\n==== Post rebuttal comments ====\nThank you for revising this work, I think it is much stronger now after the restructuring and clarification, so have increased my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the problem of learning to carry out novel, multi-task instructions specified via temporal logic using deep reinforcement learning. A specific focus of the paper is improving generalization to test-time instructions that differ from those encountered during training. To facilitate this generalization, the proposed architecture encodes a latent specification of the goal according to the given instruction and environment state that is then combined with a task-agnostic environment embedding. Experiments on grid-like domains demonstrate that the proposed framework outperforms recent deep RL approaches to satisfying temporal logic-based instructions.

The instruction-following problem has long been of interest in the robotics, ML, and broader AI communities dating back several decades. The problem has received renewed attention in the last few years, largely as a target for neural network-based multi-view and RL learning architectures. The primary contribution of this paper is the proposed extension of existing deep RL approaches to reason over a learned, latent goal specification as a means of improving generalization to novel test-time utterances. The approach is sound and several reviewers agree that the ablation studies together with comparisons to contemporary deep RL architectures support the advantage of these inductive biases. The reviewers raised initial concerns regarding the statistical significance of the results and the clarity of the presentation. The authors provided detailed feedback to the reviewers and updated the paper to address many of these concerns, largely satisfying two of the reviewers.

However, concerns remain that the paper doesn't adequately position this work in the context of the decades worth of research in instruction-following. Early work in this area focused on interpreting highly structured instructions (e.g., formal logic-based), first using rule-based methods, and then parsers trained via supervised learning. Over the past decade, however, the field has largely moved towards learning to follow instructions conveyed in "natural" language, which brings with it a significant number of challenges, including the assumption that test-time instructions will inherently be out-of-distribution. That is not to say that the contributions of the paper aren't interesting---they are, but in the relatively narrow scope of deep RL-based approaches to following structured, temporal logic instructions.
Decision: Accept (Poster)