Self-Educated Language Agent with Hindsight Experience Replay for Instruction Following
Authors: Geoffrey Cideron
Mathieu Seurin
Florian Strub
Olivier Pietquin
Venue:  (2020)
Keywords: Language
reinforcement learning
instruction following
Hindsight Experience Replay

Abstract: Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. These properties make it a natural fit to guide the training of interactive agents as it could ease recurrent challenges in Reinforcement Learning such as sample complexity, generalization, or multi-tasking. Yet, it remains an open-problem to relate language and RL in even simple instruction following scenarios. Current methods rely on expert demonstrations, auxiliary losses, or inductive biases in neural architectures. In this paper, we propose an orthogonal approach called Textual Hindsight Experience Replay (THER) that extends the Hindsight Experience Replay approach to the language setting. Whenever the agent does not fulfill its instruction, THER learn to output a new directive that matches the agent trajectory, and it relabels the episode with a positive reward. To do so, THER learns to map a state into an instruction by using past successful trajectories, which removes the need to have external expert interventions to relabel episodes as in vanilla HER. We observe that this simple idea also initiates a learning synergy between language acquisition and policy learning on instruction following tasks in the BabyAI environment. 

Under review as a conference paper at ICLR 2020
SELF-EDUCATED LANGUAGE AGENT WITH HIND- SIGHT EXPERIENCE REPLAY FOR INSTRUCTION FOL- LOWING
Anonymous authors Paper under double-bling review
ABSTRACT
Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. These properties make it a natural fit to guide the training of interactive agents as it could ease recurrent challenges in Reinforcement Learning such as sample complexity, gen- cralization, or multi-tasking. Yet, remains an open-problem to relate language and RL in even simple instruction following scenarios. Current methods rely expert demonstrations, auxiliary losses, or inductive biases in neural architectures. In this paper, we propose an orthogonal approach called Textual Hindsight Expe- rience Replay (THER) that extends the Hindsight Experience Replay approach to the language setting. Whenever the agent does fulfill its instruction, THER learns to output a new directive that matches the agent trajectory, and it relabels the episode with a positive reward. To do so, THER learns to map state into an instruction by using past successful trajectories, which removes the need to have external expert interventions to relabel episodes in vanilla HER.
INTRODUCTION
Language has slowly evolved to communicate intents, to state objectives, or to describe complex sit- uations (Kirby et al.) 2015). It conveys information compactly by relying on composition and high- lighting salient facts. Such properties are essential when developing interactive agents in complex environments. language may express vast diversity of goals and situations, could alleviate the training of interactive agents over heterogeneous and composite tasks thanks to its intrinsic struc- ture (Luketina et al. 2019). This property is all the more critical as classic Reinforcement Learning (RL) methods are facing generalization issues (Cobbe al., 2018), and learning hierarchical and structured policies remains an open- ppen-eroblem (Barto & Mahadevan, (0033) Kulkarni et al. 2016). As recently advocated by Luketina al. (2019), language should thus be considered first-class citizen to ease RL to improve on generalization and sample efficiency.
Unfortunately, conditioning policy language also entails supplementary difficulty the agent needs to understand linguistic cues to alter its behavior. The agent thus needs to ground its language understanding by relating the words to its observations, actions, and rewards before being able to leverage the language structure (Kiela et al.) 2016 Hermann et al., 2017). Once the linguistic sym- bols are grounded, the agent may then take advantage of language compositionality condition its policy on new goals. It thus leads to the following questions: are we eventually making the rein- forcement learning problem harder, or can we generate learning synergies between policy learning and language acquisition?
In this work, we use instruction following as a natural testbed to examine this question Tellex al, 2011 Chen & Mooney| [0 Artzi & Zettlemoyer 2013) Luketina et al., 2019 Zang et al.) 2018 Hermann et al. 2019 9 Chen et al., 2019).
this setting, the agent is given a text description of its goal (e.g. "pick the red ball") and is rewarded when achieving it. The agent has thus to visually grounded the language, i.c., linking and disentangling visual attributes (shape, color) from language description ("ball", "red") by using rewards to condition its policy toward task completion. On one side, the language compositionality allows for a high number goals, and offers generalization opportunities; but on the other side, it dramatically complexifies the policy search space. Besides,

---

Under review as a conference paper at ICLR 2020
Goal: Plok the red ball
purple Agente enjedtry
g: Pick the purple ball.
IP |FPoiin Trajectory
d
Instruction Datasel
Update Agent
Replay Buffer
Train Mapper
hmm.... tetched
ball.
IF Negalive Trajectory
Instruction Generator
trajectory
Pick the purple ball.
syent Eopenry f.
Plok the red ball. 1y:
Figure 1: Upon positive trajectory, the agent trajectory is used to both update the RL replay buffer and the goal mapper training dataset. Upon failed trajectory, the goal mapper used to relabel the episode, and both trajectories are appended to the replay buffer. In the original HER pa- per (Andrychowiez et al. al.] (017)). the mapping function is bypassed since they are dealing with spatial goals, and therefore, vanilla HER cannot be applied without external feedback (Chan et al. 2018).
instruction following is a notoriously hard RL problem since it has a sparse reward signal. In prac- tice, the navigation and language grounding problems are often circumvented by warm-starting the policy with labeled trajectories (Zang et al. 20188 Anderson et al., 2018). Although scalable, these approaches require numerous human demonstrations, whereas we here want to jointly learn the nav- igation policy and language understanding from scratch. In a seminal work, Hermann et al. (2017) successfully ground language instructions, but the authors used unsupervised losses and heavy cur- riculum to handle the sparse reward challenge.
In this paper, we take advantage of language compositionality to tackle the lack of reward signals. To do so, we extend Hindsight Experience Replay (HER) to language goals (Andrychowicz et al.) 2017). HER originally deals with the sparse reward problems in spatial scenario; it relabels unsuc- cessful trajectories into successful ones by redefining the policy goal posteriori. As result, HER creates additional episodes with positive rewards and a more diverse set of goals. Unfortunately. this approach cannot be directly applied when dealing with linguistic goals. As HER requires mapping between the agent trajectory and the goal to substitute, it requires expert supervision to de- scribe failed episodes with words. Hence, this mapping should either be handcrafted with synthetic bots (Chan et al.) (018), or be learned from human demonstrations, which would both limit HER generality. More generally, language adds a level of semanties, which allows generating textual ob- jective that could not be encoded simple spatial observations as in regular HER, e.g., "fetch a ball that is not blue" or "pick any red object".
In this work. we introduce Textual Hindsight Experience Replay (THER), training procedure where the agent jointly learns the language-goal mapping and the navigation policy by solely teracting with the environment illustrated in Figure 1 THER leverages positive trajectories to learn a mapping function, and THER then tackles the sparse reward problem by relabeling lan- guage goals upon negative trajectories in HER fashion. We evaluate method the BabyAI world (Chevalier- Boisvert et al. 2019), showing a clear improvement over RL baselines while high- lighting the robustness of THER to noise.
BACKGROUND AND NOTATION
In reinforcement learning, an agent interacts with the environment to maximize its cumulative re- ward (Sutton & Barto) 2018). At each time step t, the agent is in state E S, where it se- lects an action a, E A according its policy π : S + A. then receives a reward r, from the environment's 's reward function
S x A + RR and moves to the next state 8t41 with probability p($t41|s:,ar a.). The quality the policy is assessed by the Q-function defined by s, ag a] for all (s. where y [0, 1] is the discount factor. We define the optimal Q-value as Q* a) = max, (s, a). from which the optimal policy " derived. We here use Deep Q-learning (DQN) to approximate the optimal Q-function with
(*(ssa) = E, /'r(((s (rr((,),))))
2

---

Under review as a conference paper at ICLR 2020
ral networks and perform off off-poliey updates by sampling transitions (st, de, Me, St+1) from replay buffer (Mnih et al., 2013).
In this article, augment our environment with goal space G which defines new reward function r : S x A x G + R. and policy т : S x G - A by conditioning them on a goal descriptor g G. Similarly, the Q-function is also conditioned Value Function Approximator (UVFA) (Schaul et al.] (015). This approach allows learning holistic policies that generalize over goals in addition to states the expense of complexifying the training process. In this paper, we explore how language can be used for structuring the goal space, and how
the goal, and it is referred to Universal
language composition eases generalization over unseen scenarios in a UVFA setting.
Hindsight Experience Replay (HER) (Andrychowicz et al. 2017) is designed to increase the sample efficiency of off-policy RL algorithms such as DQN in the conditionnn setting. reduces the sparse reward problem by taking advantage of failed trajectories, relabelling them with new goals. An expert then assigns the goal that was achieved by the agent when performing its trajectory, before updating the agent memory replay buffer with an additional positive trajectory.
Formally, HER assumes the existence of a predicate J S x G + 1) which encodes whether the agent in state satisfies the goal f(s, g) = 1, and defines the reward function r(st, a. f 9). At the beginning of an episode, a goal g is drawn from the space G of goals. At each time step f, the transition (st, ax, Ta, 9)) is stored in the DQN replay buffer, and at the end of an unsuccessful episode, an expert provides an additional goal g' that matches the trajectory. New transitions (St; de, r2: , are thus added the replay buffer for each time step I, where r' = rrtt, Oe, $t41,9"). DQN update rule remains identical to (Mnih et al., 2015). transitions are sampled from the replay buffer, and the network is updated using one step to id-error minimization.
HER assumes that a mapping m between states s and goals g is given. In the original pa- per (Andrychowicz et al. 2017). this requirement is not restrictive the goal space subset of the state space. Thus, the mapping m is straightforward since any state along the trajectory can be used as substitution goal. In the general case, the goal space differs from the state space, and the mapping function is generally unknown. In the instruction following setting, there is no obvious mapping from visual states to linguistic cues. It thus requires expert intervention to provide new language goal given the trajectory, which drastically reduces the interest of HER. Therefore, we here explore how to learn this mapping without any form of expert knowledge nor supervision.
TEXTUAL HINDSIGHT EXPERIENCE REPLAY
Textual Hindsight Experience Replay (THER) aims to learn a mapping from past experiences that relates trajectory to goal in order to apply HER, even when no expert are available. The mapping function relabels unsuccessful trajectories by predicting a substitute goal 9 as an expert would do. The transitions are then appended to the replay buffer. This mapping learning is performed alongside agent policy training.
Besides, we wish to discard any form of expert supervision to learn this mapping as it would reduce the practicability of the approach. Therefore, the core idea is to use environment signals to retrieve training mapping pairs. Instinctively, in the sparse reward setting, trajectories with positive rewards encode ground-truth mapping pairs, while trajectories with negative rewards are mismatched pairs. These cues are thus collected train the mapping function for THER in supervised fashion. We emphasize that such signals are inherent to the environment, and an external expert does not provide them. In the following, we only keep positive pairs in order to train a discriminative mapping model.
Formally, THER is composed of a dataset D of (8,g) g) pairs, a replay buffer I and a parametrized mapping model mmm- For each episode, goal g is picked, and the agent generates transitions (st, de. Tr, St+1: g) that are appended to the replay buffer R. The Q-function parameters are updated with an off-policy algorithm by sampling minibatches from D. Upon episode termination, if the goal is achieved, i.c. f(r,gg g) = 1, the (ST, g) pair is appended to the dataset D. If the goal is not achieved, a substitute goal is sampled from the mapping mode [1 (sT) = g' and the additional transitions {st, az, Ft, $t+1. o') T are added to the replay buffer. At regular intervals, the mapping
'The mapping model can be utilized with accuracy criterion over validation set to avoid random goal sampling. see algorithh T for more details
3

---

Under review as a conference paper at ICLR 2020
Trajectory
Pick the purple ball
MLP
state embedding
goal embedding
RNN (decoder)
RNN (encoder)
RNN
state embedding
Conv
Pick the purple ball
[s 9g
Figure 2: Left: Models are fed with the pre-extracted observations from the trajectories. Middle: Agent model whose inputs the last four observations and the goal. Right: Instruction Generator.
model m is optimized to predict the goal given the trajectory r by sampling mini-batches from D. Algorithm summarizes our approach. Noticeably, THER can be extended to partially observable environments by replacing the predicate function f(, by f(T, g). i.e., the completion of a goal depends on the full trajectory rather than one state. Although we assess THER in the instruction following setting, the proposed procedure can be extended to any other goal modalities.
EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Environment We experiment our approach on grid world environment called MiniGrid (Chevalier- Boisvert et al. 2019). This environment offers a variety of instruction-following tasks using a syn- thetic language for grounded language learning. We use 10x10 grid with 10 objects randomly located in the room. Each object has 4 attributes (shade, size, color, and type) inducing a total of 300 different objects. The agent has four actions {forward, left, right, pick), and it can only the 7x7 grid in front of it. For each episode, one object's attribute is randomly picked as goal, and the text generator translates it in synthetic language detailed in Appenddix C. e.g.. "Fetch a tiny light blue ball." The agent is rewarded when picking one object matching the goal description. which ends the episode; otherwise, the episode stops after 40 steps or after taking an incorrect object.
Models In this experiment, THER is composed of separate models shown in Figure 2 The instruction generator is a neural network outputting sequence of words given the final state of a trajectory. It is trained by gradient descent using a cross-entropy loss on the dataset D collected as described in section 3 We train DQN network following (Mnih et al.) (2015) with a dueling head (Wang et al. (2066), double Q-learning (Hasselt et al. 2016), and uniform replay buffer. The network receives a tuple < s.g as input and output an action corresponding to the argmax over states-actions values (ss, a, .). We use e-greedy exploration with decaying e. The detailed models and hyperparameters are provided in Appendix A. and the source code is available at HIDDEN FOR REVIEW
4.2 BUILDING INTUITION
This section examines the feasibility of THER by analysing two potential issues. We first show that HER is robust to noisy mapping function (or partially incorrect goals), then estimate the accuracy and generalisation performance of the instruction generator.
Noisy instruction generator and HER
We investigate how a noisy mapping m affects perfor- mance compared to perfect mapping. As the learned instruction generator likely be imperfect, is crucial to assess how a noisy mapping may alter the training of the agent. To do so, we train an agent with HER and synthetic bot relabel unsuccessful trajectories. We then inject noise
4

---

Under review as a conference paper at ICLR 2020
0.4
0.8
23 2.
0.3
0.0
0.2
9.0
500
1000
1500
5000
1000C
15000
length of the treninng 19t
0.0
0.5
1.0
Figure 3: Left: Agent performance with noisy mapping function. Right: Instruction generator ac-
curacy over 5k pairs. Figures are averaged over seeds and error bars shows one standard deviation.
in our mapping where each attribute has a fixed probability p to be swapped, e.g. color blue may be changed green. For example, when p 0.2, the probability of having the whole instruction correct is 0.84 s 0.4. The resulting agent performance is depicted in Figure 3 (left).
The agent performs 80% as well as an agent with perfect expert feedback even when the mapping function has a 50% noise-ratio per attribute. Surprisingly, even highly noisy mappers, with a 80% noise-ratio, still provides an improvement over vanilla DQN-agents. Hence, HER can be applied even when relabelling trajectories with partially correct goals.
We also examine whether this robustness may be induced by the environment properties (e.g. tribute redundancy) rather than HER. We thus compute the number of discriminative features re- quired to pick the correct object, as shown Figure 8] On average, an object can be discriminated with 1.7 features in our setting which eases the training, but any object shares at least one property with any other object 70% of the time- which tangles the training. Besides, the agent does not know which features are noisy or important. Thus, the agent still has to disentangle the instructions across trajectories in the replay buffer, and this process is still relatively robust to noise.
Learning an instruction generator We briefly analyze the sample complexity and generalization properties of the instruction generator. If training the mapping function is straightforward than learning the agent policy, then we can thus use it to speed up the navigation training.
We first split the of missions G into two disjoint sets Gtrain and Grest- Although all object features are present in both sets, they contain dissimilar combinations of target objects. For in- stance, blue, dark, key, and large are individually present in instructions of Gtroin and Grest but the instruction to get large dark blue key is only in Great- We therefore assess whether a basic compositionality is learned. the following, we use train/split ratio of 80/20, i.e.. 240 vs 60 goals.
Finally, we generate an artificial dataset D of (g. ST) pairs, and we report the training/testing accu- racy of the instruction generator Figure (right). The accuracy evaluates whether the four correct attributes are present in the linguistic instructions through a simple parser. For instance, "a large blue light key" is a failure case since one attribute is missing. Note that language accuracy is discussed further in subsection 4.4 Other language metrics can be used when dealing with natural language like BLEU (Papineni et al. 2002). ROUGE (Ln 2004). METEOR (Banerjee & Lavie 2005).
We here observe than 1000 positive episodes necessary to reach around 20% accuracy with model, and 5000 pairs are enough to reach 70% accuracy. The instruction generator also correctly predicts unseen instructions even with fewer than 1000 samples and the accuracy gap between and unseen instructions slowly decrease during training, showing basic compositionality acquisition. As further discussed in /section 5] we here vanilla mapping architecture to assess the generality of our THER, and more advanced architectures may drastically improve sample complexity (Bah- danau et al. 2019b).
4.3
THER FOR INSTRUCTION FOLLOWING
In the previous section, we observe that: (1) HER robust to noisy relabeled goals, (2) an instructor generator requires few positive samples learn basic language compositionality. We thus here
5

---

Under review as a conference paper at ICLR 2020
0.5
Dofeeeeerr shaping
0.4
1924
0.6
Successs rate
0.3
accuracy
10.4
0.2
0.1
0.0
0.0
5
environment steps
environment
te5
Figure 4: Left: learning curves for DQN, DQN+HER, DON+THER in 10x10 gridworld with 10 objects with attributes. The instruction generator is used after the vertical bar. Right: the mapping accuracy for the prediction of instructions. mww starts being trained after collecting 1000 positive trajectories. Results are averaged over 5 seeds with one standard deviation.
combine those two properties to execute THER, i.e. jointly learning the agent policy and language prediction in a online fashion for instruction following.
Baselines We want to assess if the agent benefits from learning an instruction generator and us- ing it to substitute goals as done in HER. We denote this approach DON+THER. We compare approach to DQN without goal substitution (called DQN) and DQN with goal substitution from a perfect mapping provided by external expert (called DQN+HER) available in the Baby AI envi- ronment. We emphasize again that it is impossible to have an external expert to apply HER in the general case. Therefore, DQN is lower bound that expect outperform, whereas DQN+HER is the upper bound as the learned mapping can not outperform the expert. Note that we only start using the parametrized mapping function after collecting 1000 positive trajectories, which is around 18% validation accuracy. Finally, we compute an additional DQN baseline denoted DQN+reward: we reward the agent with 0.25 for each matching properties when picking a object given an instruction. It enforces a hand-crafted curriculum and dramatically reduces the reward sparsity, which gives different perspective on the current task difficulty.
Results In Figure (left), show the success rate of the benchmarked algorithms per environ- ment steps. We first observe that DQN does not manage to learn a good policy, and its performance remains close that of random policy. On the other side, DQN+HER and D quickly manage to pick the correct object 40% of the time. Finally, DQN+THER sees its success rates in- creasing as soon as we use the mapping function, rapidly perform nearly as well as DQN+HER. Figure 4 (right) shows the performance accuracy of the mapping generator by environment steps. We observe steady improvement of the accuracy during training before reaching 78% accuracy after 5M steps. In the end, DQN+THER outperforms DQN by using the exact same amount of in- formation, and even matches the conceptual upper bond computed by DQN+HER. Besides, THER does not alter the optimal policy which can occur when reshaping the reward (Ng et al. 1999).
Discussion As observed in the previous noisy-HER experiment, the policy success rate starts increasing even when the mapping accuracy 20%, and DON+THER becomes nearly good as DON+HER despite having maximum mapping accuracy of 78%. It demonstrates that DON+THER manages to trigger the policy learning by better leveraging environment signals com- pared to DQN. As the instruction generator focuses solely on grounding language, it quickly pro- vides additional training signal to the agent, initiating the navigation learning process.
We observe that the number of positive trajectories needed to learn a non-random mapping m.w lower than the number of positive trajectories needed to obtain a valid policy with DQN (even after 5M environment steps the policy has 10% success rate). Noticeably, we artificially generate dataset in section 4.2 to train the instruction generator, whereas we follow the agent policy to collect the dataset, which is more realistic setting. For instance, the instructor generator is trained on a moving dataset, it could overfit to the first positive samples, but in practice it escapes from local minima and obtains final high accuracy.

---

Under review as a conference paper at ICLR 2020
ireeevant words
correct wonds
3.5
2.5
2.0
1.5
1.0
0.5
0.0
2.
10
25
200
500
730
bungl of daaining sol
1000
1500
2000
1000
10000 15000
Figure 5: Left: Transition distributions in the replay buffer between successful, unsuccessful and relabeled trajectories. We remove time-out trajectories for clarity. which accounts for 54% of the transition average (+-3% over training). Right: Evaluating the language learned by the instruction generator on unseen instructions. Over time, the number of correct attributes (purple) is increasing, as the number of irrelevant words (orange) is decreasing. The number of repeated attributes (green) stays low. The beginning clause is ignored as it doesn't provide information regarding the objective.
Different factors may also explain the learning speed discrepancy: supervised learning has less vari- ance than reinforcement learning as it has no long-term dependency. The agent instructor generator can also rely on simpler neural architectures than the agent. Although THER thus takes advantage of those training facilities to reward the agent ultimately.
Finally, we observe virtuous circle that arises. As soon as the mapping correct, the agent suc- cess rate increases, initiating the synergy. The agent then provides additional ground-truth mapping pairs, which increases the mapping accuracy, which improves the quality of substitute goals, which increases the agent success rate further more. As result, there is a natural synergy that occurs be- tween language grounding and navigation policy as each module iteratively provides better training samples to the other model. This virtuous circle is observed inside the replay buffer distribution, as shown in Figure 5] we ignore time-out trajectories, around 90% of the trajectories are nega- tive the beginning of the training. As soon transitions are relabelled by the instructor generator, and 10% of the transitions belong to positive trajectories. As training goes, this ratio slowly inverted, and after 5M steps, there only 15% relabelled trajectories left while 60% are actual positive trajectories.
start using the instruction generator, 40% the
Limitations Albeit generic, THER also faces some inherent limitations. From linguistic per- spective, THER cannot transcribe negative instructions (Do not pick the red ball), or alternatives (Pick the red ball or the blue key) in its current form. However, this problem could be alleviated by batching several trajectories with the same goal. Therefore, the model would potentially learn to factorize trajectories into single language objective. On the policy side, THER still requires a few trajectories to work, and thus relies on the navigation policy. In other word, historical HER could be applied in the absence of reward signals, while THER only alleviate the sparse reward problem by better leveraging successful trajectories. A natural improvement would be to couple THER with other exploration methods, e.g, intrinsic motivation (Bellemare al. 2016 DQN with human demonstration (Hester et al. 2018). Finally, under-traines goal generators might hurt the training in some environments although we did observed it in our setting as shown in Figure 9 However, simple validation accuracy allows to circumvent this risk while activating the goal mapper (More details in algorithm T}. We emphasize again that the instruction generator can be triggered anytime to kick-start the learning as the it is independent of the agent.
4.4 LANGUAGE LEARNED BY THE INSTRUCTION GENERATOR
We here amalyze further the language quality of the instruction generator. To do so, we rely on three metrics to assess the generated language quality. The first metric, called attribute fidelity, assesses whether every target attribute is present in the generated sentence. For example, for the objective a large dark blue key, the generated sentence "Fetch me large key" only containing two attributes and receives a score of two. Yet, the model may still enumerate all available attributes in a single sentence; the score would always be four. Language precision counter this effect by counting how
7

---

Under review as a conference paper at ICLR 2020
Instruction
get small very very- Jight green key
get a tiny dark yellow key
go letch dark grey giant ball
#Samples
200
get neutral very_light tny ball
get small blue ball
go get a grey giant neutral giant neutral grey
1000
get very_light green small ball
go fetch tiny dark tiny ball
you must fetch a grey dark giant ball go fetch a grey dark gian ball
10000
gel a very Jight green small key
go get dark yellow tiny key
Table 1: Examples of language errors during the training
many words are not relevant to describe the target object. This second metric is related to precision (or positive predictive value), as generated instructions only contain relevant attributes. Finally, we count repeated attributes as language models are known to stutter during early training.
In Figure 5
we compute the three metries over unseen goal states, examining the compositionality properties of the instruction generator. We observe that generated instructions get more accurate, contain less irrelevant attributes, thus providing the agent with valid goals, even in unseen scenarios. <state, instruction> pairs col-
As the instruction generator is trained until convergence lected, it naturally preserve the overall language structure, and correctly ground symbols: repeated attributes score remains low and generated sentences start with the verb and end with the noun while randomly shuffling the attributes as shown in Table I
RELATED WORK
Instruction following have recently drawn a lot of attention following the emergence of several 2D and 3D environments Sievalier-Boisved 2018). This section first provides an overview of the different approaches. i.c. fully-supervised agent, reward shaping, auxiliary losses, before exploring approaches related to THER.
al., 2019 Brodeur et al.) 2017 Anderson et al.
Vision and Language Navigation Instruction following is sometimes coined as Vision and Lan- guage Navigation tasks in computer vision (Anderson al., 2018 Wang al. 2019). Most strate- gies are based on imitation learning, relying on expert demonstrations and knowledge from the envi- ronment. For example, Zang et al. (2018) relate instructions to an environment graph, requiring both demonstrations and high-level navigation information. Closer to our work, Fried et al. (2018) also learns a navigation model and an instruction generator, but the latter is used to generate additional training data for the agent. The setup is hence fully supervised, and requires human demonstrations. These policies are sometimes finetuned to improve navigation abilities in unknown environments. Noticeably, Wang et al. (2019) optimizes their agent to find the shortest path by leveraging language information. The agent learns an instruction generator, and they derive an intrinsic reward by align- ing the generator predictions over the ground truth instructions. Those approaches complete long sequences of instructions in visually rich environments but they require a substantial amount of an- notated data. In this paper, we intend to discard human supervision to explore learning synergies. Besides, we needed a synthetic environments with experts to evaluate THER. Yet, THER could be studied natural and visually rich settings by warm-starting the instruction generator, and those two papers give a hint that THER could scale up to larger environment.
IRL for instruction following Bahdanau et al. (earr) mapping from <instruction, state>> to a reward function. The method's aim is to substitute the environment's reward function when instructions can be satisfied by great diversity of states, making hand-designing reward function tedious. Similarly, [u et al.] (2019) directly learn a reward function and assess its transfcrability to new environments. Those methods complementary to ours they seek to transfer reward function to new environment and we are interested in reducing sample complexity.
Improving language compositionality THER heavily relies on leveraging the language struc- ture in the instruction mapper toward initiating the learning synergy. For instance, [Bhdanau et al.) (2019b) explore the generalization abilities of various neural architectures. They show that the sam- ple efficiency of feature concatenation can be considerably improved by using feature-wist modu- lation (Perez et al. 2018). neural module networks (Andreas et al. 2016) or compositional attention networks (Hudson & Manning 2018). In this spirit, Bahdanau et al. (2019a) take advantage of these architectures to quickly learn dense reward model from a few human demonstrations in the instruc-
8

---

Under review as a conference paper at ICLR 2020
tion following setup. Differently, the instructor generator can also be fused with the agent model to act as an auxiliary loss, reducing further the sparse reward issue.
HER variants HER has been extended to multiple settings since the original paper. These exten- sions deal with automatic curriculum learning (Liu et al.) 2019), dynamic goals (Fang et al. 2019). or they adapt goal relabelling to policy gradient methods (Rauber et al., 2019). Closer to our work, Sahni et al. (2019) train a generative adversarial network to hallucinate visual near-goals state over failed trajectories. However, their method requires heavy engineering as visual goals are extremely complex to generate, and they lack the compact generalization opportunities inherent to language. Chan et al. (2018) also studies HER in the language setting, but the authors only consider the context where a language expert is available.
Conditioned Language Policy There have been other attempts to leverage language instruction improve the agent policy. For instance, Diang al. (2199) computes a high-level language policy to give textual instruction to low-level policy, enforcing a hierarchical learning training. The authors manage to resolve complicated manipulating task by decomposing the action with language operation. Yet, the language mapper performs instruction retrieval into a predefined set of textual goals, which prevent from benefiting from language compositionality, as mentioned by the authors. Co-Reyes et al. (2018) train agent to refine its policy by collecting language corrections over multiple trajectories on the same task. While the authors focus their effort on integrating language cues, it could be promising learn the correction function in THER fashion.
6 CONCLUSION
We introduce Textual Hindsight Experience Replay (THER) as extension to HER for language. We define a protocol to learn a mapping function to relabel unsuccessful trajectories with predicted consistent language instructions. We show that THER nearly matches HER performances despite only relying on signals from the environment. We provide empirical evidence that THER manages to alleviate the instruction following task by jointly learning language grounding and navigation policy with training synergies. THER has mild underlying assumptions, and it does not require human data, making it valuable to complement other instruction following methods. More generally, THER can be extended to any goal modalities, and we expect similar procedures to emerge in other setting.
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf,, Ian Reid, Stephen Gould, and Anton van den Hengel. iision-anddlangguae navigation: Interpreting visually-grounded navigation instructions in real environments. In Proc. of CVPR, 2018.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proc. of CVPR, 2016.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience play. In Proc. of NIPS, 2017.
Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49-62, 2013.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. In Proc. of ICLR, 2019a.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can be learned? In Proc. of ICLR, 2019b.

---

Under review as a conference paper at ICLR 2020
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proc. of ACL Workshop on Intrinsic and Extrinsic Evalua- tion Measures for Machine Translation and/or Summarization, 2005.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(1-2):41-7, 2003.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos.
Unifying count-based exploration and intrinsic motivation. In Proc. of NeurIPS, 2016.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Proc. of NIPS, 2015.
Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron Courville. Home: a household multimodal environment. In Proc. of NIPS Visually-Gruudded Interaction and Language Workshop, 2017.
Harris Chan, Yuhuai Wu, Jamie Kiros, Sanja Fidler, and Jimmy Ba. Actree: Augmenting experience via teacher's advice for multi-goal reinforcement learning. 1st Workshop on Goal Specifications for Reinforcement Learning, Workshop held jointly at ICMI, IJCAI, AAMAS, 2018
David I. Chen and Raymond Mooney. Learning to interpret natural language navigation instruc- tions from observations. In Proc. of AAAI, 2011.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proc. of CVPR, 2019.
Maxime Chevalier-Boisee
Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. BabyAl: First steps towards grounded language learning with human in the loop. In Proc. of ICLR, 2019. URL httts::/ net
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In Proc. of NIPS Workshop on Deep Learning, 2014.
John D Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, John DeNero, Pieter Abbeel, and Sergey Levine. Guiding policies with language via meta-learning. Proc. ICLR, 2018.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning, arXiv preprint arXiv: 1812.0231, 2018.
Meng Fang, Cheng Zhou, Bei Shi, Boqing Gong, Jia Xu, and Tong Zhang. Dher: Hindsight experi- ence replay for dynamic goals. In Proc. of ICLR, 2019.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg- Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for lnn-lang navigation. In Proc. of NeurIPS, 2018.
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. In Proc of. ICLR, 2019.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proc. of AAAI, 2016.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czamecki, Max Jaderberg, Denis Teplyashin, et al. Grounded lan- guage learning in a simulated 3d world. arXiv preprint arx/v.1006.06551 2017.
Karl Moritz Hermann, Mateusz Malinowski, Piotr Mirowski, Andras derson, and Raia Hadsell. Learning to follow directions in street view. arXiv preprint arXiv: (50::/99300044 2019.
Keith An-
10

---

Under review as a conference paper at ICLR 2020
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proc. of AAAI, 2018.
Drew Arad Hudson and Christopher D. Manning. Compositional attention networks for machine reasoning. In Proc. of ICLR, 2018.
Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language an abstraction for hier- archical deep reinforcement learning. arXiv preprint arXiv: meter/906007333. 07343, 2019.
Douwe Kiela, Luana Bulat, Anita I. Vero, and Stephen Clark. Virtual embodiment: A scalable long-term strategy for artificial intelligence research, arXiv preprint arXiv: 2016.
Diederik P Kingma and Jimmy Lei Ba. Adam: Amethod for stochastic optimization. In Proc. of ICLR, 2015.
Simon Kirby, Monica Tamariz, Hannah Corish, and Kenny Smith. Compression and communica- tion in the cultural evolution of linguistic structure. Cognition, 141:87-102, 2015.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Proc. of NIPS, 2016.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 1995.
Chin-Yes Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004.
Hao Liu, Alexander Trott, Richard Socher, and Caiming Xiong. Competitive experience replay. In Proc. of ICLR, 2019.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. In Proc. of IJCAI, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 518(7540):529, 2015.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proc. of ICML, 1999.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proc. of ACL, 2002.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proc. of AAAI, 2018.
Paulo Rauber, Avinash Ummadisingu, Filipe Mutz, and Jrgen Schmidhuber. Hindsight policy gra- dients. Proc. of ICLR, 2019.
Himanshu Sahni, Toby Buckley, Pieter Abbeel, and Ilya Kuzovkin. Visual hindsight experience replay. arXiv preprint arXiv: 1901.115299 2019.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In Proc. of ICML, 2015.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In Proc. of NIPS, 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
11

---

Under review as a conference paper at ICLR 2020
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proc. of AAAI, 2011.
Xin Wang. Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang. William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imi- tation learning for vision- lasion-lngguag navigation. In Proc. of CVPR, 2019.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In Proc. of ICML, 2016.
Xiaoxue Zang, Ashwini Pokle, Marynel Vázquez, Kevin Chen, Juan Carlos Niebles, Alvaro Soto, and Silvio Savarese. Translating navigation instructions natural language to a high-level plan for behavioral robot navigation. In Proc. of EMNLP, 2018.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX A: TRAINING DETAILS
A.1 INSTRUCTION GENERATOR ARCHITECTURE
The encoder is convolution neural network (LeCun al. 1995) with ReLU activation functions after each layer for processing an observation of dimension 7x7x3. It is composed of: 16 of 2x2 kernel, followed a max pooling 2D of size 2x2 then 32 of 2x2 kernel, and finally 256 of 2x2 kernel. For the convolutional layers and the pooling layer, the stride is always equal to 1.
The decoder is a recurrent neural network composed of gated recurrent units (GRU) CChung et al, 2014) that outputs the instruction word by word. The convolutional layers extract relevant informa- tion from the image and compress in a latent representation. This latent representation then used as the initial hidden state of GRU layer. The initial input of the GRU layer is the token BEGIN. At each time step, the GRU layer outputs a distribution over words as in Seq2Seq (Sutskever et al., 2014, we can use as the input of the next step the word the is the most likely. When the token END is chosen the prediction stops. We use a word embedding of dimension 128 and an instruction embedding of dimension 256. The number of words in the dictionary equal 27.
The training is done with cross-entropy between the distribution of probabilities over words pre- dicted by the model and the true word. Teacher forcing employed accelerate and stabilize the learning. Teacher forcing refers to using the ground truth for the next input of the GRU instead of using the last predicted word (Bengio et al., 2015).
The learning rate is fixed to 10-1 and the network is trained using the Adam optimizer with default parameters (Kingma & Ba 2015) and regularization of 10-6 over parameters. The batchsize equal to 128.
A.2 DQN ARCHITECTURE
To deal with the partial observability of the environment (described in subsection 4.1 the state corresponds to the last 4 frames stacked as in (Mnih al. 2015).
Visual Encoder: Each frame is encoded by convolutional neural network and then passed through LSTM. Each layer is followed by a ReLU activation function. The convolutional neural network is composed of: 16 of 2x2 kernel, followed a max pooling 2D of size 2x2 then 32 of 2x2 kernel, and finally 64 of 2x2 kernel. For the convolutional layers and the pooling layer, the stride is always equal to 1. The LSTM has an input and hidden size of 64. The last LSTM hidden state h, corresponds to visual embedding.
Instruction Encoding: First each word is embedded with an embedding of size 32. Instructions are encoded word by word using GRU. The GRU has an input size of 32 which corresponds to the word embedding size and hidden size of 128 which corresponds to the instruction embedding size. The last GRU hidden state h2 is kept as the instruction embedding.
We concatenate the visual embedding and the instruction embedding and add fully connected layers in the same fashion as in the dueling architecture (Wang et al. 2016) on top to compute the Q-values for each action.
Training parameters The exploration policy is e-greedy with e decaying linearly from to 0.05 either in 500 000 steps for DQN and DQN+THER or in 100 000 steps for DON+HER. The R.Mprrpp optimizer used to train the neural network with a learning rate fixed to 10 and default parameters
(the forgetting factor is 0.99). Huber loss (with 8 1) and gradient clipping (between -I and 1) are used for stable gradients. The target network is synchronized with the current model every 1000 gradient steps. The replay buffer size is 50 000.
13

---

Under review as a conference paper at ICLR 2020
B THER ALGORITHM DETAILED
Algorithm 1: Textual Hindsight Experience Replay (THER)
Given:
an off-policy RL algorithm (e.g. DQN) A
reward function S A +
a language score (e.g. parser accuracy, BLEU ct..
Initialize &, replay buffer R, dataset Divain and Deat of (instruction, state), Instruction Generator mw: for episode=I,M
Sample goal and an initial state 80;
== 0:
repeat
Execute an action a, chosen from the behavioral policy az < (s:119)):
Observe reward Fi = r(St, at,g) 9) and a new state $t41i
Store the transition (st, $t-1, g) in Rt: Update Q-network parameters using the policy A and sampled minihatches from R;
t ==
11
until episode ends; if (*t, then
Store the pair (8 in Derain Deali Update m-network parameters by sampling minibatches from Darain; end else
14
17
m language validation score is high enough and Dont big enough then Sample ÿ' = Men [ss):
18
Replace g by ÿ' the transitions of the last episode and set r(se; ae, g').
20
end
end
22 end
14

---

Under review as a conference paper at ICLR 2020
C APPENDIX B: ENVIRONMENT DETAILS
screenshot of the environment is provided Figure
The state of the environment does not correspond to RBG image but to channels encoding different info about each cell (color, type etc.... More details are available at gym-minigrid
The synthetic language used for instructions is composed in three parts. First, clause like gel (all the clauses are displayed below). Then, series of attributes randomly ordered describing the object and lastly, the type of the object. An example of instruction is Go fetch tiny dark red ball. One object can be described by a maximum attributes: shade, size, color, and type. They have modalities going from 2 6.
Instruction: go fetch purple ball
Figure 6: A screenshot of the environment. The agent only sees the light gray area. should be noted that in is this example, object only have 2 attributes (color and type) but in our experiments, attributes are being used.
The environment used is grid-world of variable size containing objects. Objects can be described using up to 4 attributes. The list of all attributes is the following:
Shapes key, ball
Colors red, green, blue, purple, yellow, grey
Shades very dark, dark, neutral, light, very light
Sizes tiny, small, medium, large, giant
The five possible clauses are:
get
go fetch a
get
fetch a
you must fetch
Multiple possible clauses bring diversity to the language as their meaning is equivalent.
D COMPLEMENTARY EXPERIMENT
N-gram measure An n-gram is sequence of words, e.g. 2-gram corresponds to two-word sequence. For example the sentence Get a red ball. is composed of three 2-gram: Get a, a red, red ball and two 3-gram: get red and red ball. The n-gram measure assesses the language model accuracy by counting how many n- n-grams in the original sentence is present in the generated one. This measure close BLEU score used in machine translation (Papineni al. 2002).
15

---

Under review as a conference paper at ICLR 2020
In experiments, the language used is synthetic, and attributes order is random. Therefore, the attributes' presence is more important than the position each word. To assess the learned language accuracy, compare generated sequences to what call randomized ground truth. Comparing generated instruction to instructions generated by the environment is not relevant as the beginning clause (i.e., Get Fetch a, etc.) and attributes order random. Therefore, for given ground truth instruction, object attributes are shuffled in the sentence, and the beginning clause is sampled from all possible clauses. Since the beginning clause is random, even randomized ground truth cannot reach an accuracy of 1. The lower bound called Random Attributes corresponds to sampling clause and each attribute randomly.
predictions random atribuues
0.7
match percentage
0.6
0.3
0.0 1-gram
2-gram
3-gram
4-gram
Figure 7: Quality of the language learned by the instruction generator with 10 000 samples. For randomized ground truth the sentences are the same as the ones from the ground truth but the order of the attributes is shuffled and the clause is changed. For random attributes beginning clause and object attributes are picked randomly.
Figure 7 shows that the language learned by the instruction generator is close to the upper bound randomized ground truth. These results correlate with/Figure (right), indicating that the instruction generator can produce instruction containing correct object attributes. The fast decrease in accuracy when n grows can be explained by the attributes order randomness.
04
05
0/5
02
011
Figure 8: Left: Number of shared attributes between two objects. Right: The number of attributes needed discriminate an object from all the others.
16

---

Under review as a conference paper at ICLR 2020
DQN+HER
DON+THER started scratch
0.4
DON+THER started 1000 samples
DON+THER started 2000 samples
success rate
DQN
0.3
0.2
0.1
0.0
1
2
4
environment steps
le6
Figure 9: The instruction generator is triggered after collecting O, 1000 and 2000 positive trajecto-
ries, i.e, approximately 0%, 20%, 50% accuracy following /eccionn 4.2 Even when the instruction generator is not accurate, the policy still makes steady progress and the final success rate is impacted. Delaying the generator instructor does not provide additional benefit
sellowngssection
17


Reviews:
[{"review_id": "Hyx1kN-K_r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes THER (textual hindsight experience replay), which extends the HER algorithm to the case when goals are represented as language. Whereas in HER the mapping from states to goals is the identity function, THER trains a separate modeling network which performs a mapping from states to goals represented by language. The policy (represented as a Q-function trained via DQN) takes in the goal (a command) as an additional argument as done in HER, which allows the agent to be commanded different tasks. The authors evaluate THER on the MiniGrid environment, where they demonstrate that THER greatly outperforms vanilla goal-conditioned DQN, even in the presence of significant label noise.\n\nOverall, combining HER with language-based goals is an interesting and novel problem, and potentially a promising approach to solving language-conditioned reinforcement learning where sparse rewards are common. The authors show fairly convincingly that THER heavily outperforms DQN, which fails to improve from the random initial policy. However, I have several conceptual concerns with the proposed algorithm:\n\n1) There seems to be a bootstrapping problem in the algorithm, with regards to the instruction generator and the policy. If the algorithm does not succeed in reaching goals, then the instruction generator m_w has little training data. However, if m_w is not good, then the algorithm will not be able to give good reward signal to the policy. HER does not have this problem as it has an oracle goal mapping function, so m_w is always good. Evidently, the algorithm worked on the domain that it was tested in, but do the authors have any intuition on when this bootstrapping behavior could be harmful, or some justification on why it would not happen? If the language was more complex (and not limited to a small set of template instructions), would the THER approach still be reasonable?\n\n2) How does the algorithm detect if a given goal state corresponds to successful execution of a command? Or in the notation of the paper, how is f(s,g) implemented? In general, this does not seem like a trivial question to answer if one were to implement this algorithm in a real-world scenario.\n\nMy overall decision is borderline (learning towards accept), as the experiments were well done and serve as a good proof-of-concept, but I am unsure if this approach will scale well outside of the particular tested domain.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "ByxCLQLRFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "What is the specific question/problem tackled by the paper?\nThis paper tackles the problem of learning language-conditioned policies from reinforcement learning. Unlike most language-conditioned navigation work which relies on human demonstrations (e.g. in the room2room environment), this work only learns from the agent’s experience using a generalization of hindsight experience replay.\n\nMethod overview:\nTHER (textual HER) generalizes HER to cases where goals are not in the same space as states. To deal with this gap, THER learns a mapping from state space to goal space using successful trajectories. This mapping is then used to relabel unsuccessful trajectories with a guess of what goal was reached. This intuitive approach allows the text-conditioned agent to reach  40% at a 2D navigation task when conditioned on text such as “Pick the large red circle”. \n\nStrengths:\nThe method is well motivated and would be useful. The ablations of showing how many successful trajectories are needed to learn the mapping (~1000-5000), how many time steps are needed to reach 1000 successes (~400k steps), and how accurate the mapping needs to be for HER to work (~80%) and thorough and easy to understand. This experimental completeness is itself a contribution. \n\nAdditionally, although the authors do not discuss this, this method is actually agnostic to the particular modality (e.g. text) of the goal space and could be used anytime the goal space differs from the state space. \n\nWeaknesses:\nThe primary weakness of the paper is that the testbed environment and the textual goals are very simple. The “language” is just a list of up to 4 attributes describing the different objects and the control is simple navigation without any walls of visual variation. Additionally, the method requires accidentally getting successful trajectories early in training in order to train the mapping, and in this environment it is very easy to get successful trajectories. \nI would interested in seeing how this method would work in the room2room environment (or some other more complex task). While it is unlikely to outperform the prior methods that use the human demonstrations, it would be useful to see how close THER can get to that performance and with how many environment steps. The advantage of this environment is that it has real human knowledge, and the textual goals are limited in number, making the experiment much more realistic (as humans are unlikely to sit next to an agent and generate infinitely many diverse textual goals). \n\nA missing ablation in Figure 4 left is THER without waiting 400k steps before relabeling. In realistic scenarios, we would not be able to evaluate the mapper ahead of time to know when to start relabeling. How is performance affected is this knowledge is not available?\n\nOverall, I lean to reject the paper in it’s current form, I believe this paper would be more impactful with experiments involving more language complexity or more policy complexity.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJlpIE-6tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work attempts to learn instruction following agents without the requirement for a paired instruction, behavior corpus and instead simply relying on environment interaction and using a form of hindsight relabeling to learn how to relate language and behavior. \n\nIntroduction: \n\nIt thus leads to the following questions: are we eventually making the rein-forcement learning problem harder, or can we generate learning synergies between policy learningand language acquisition? -> it really doesn’t seem like the point of instruction following is that. It seems like you want instruction following so that you can communicate novel objectives to your agent, with the promise of generalization. Maybe reword?\n\nThe motivation for using HER makes sense, but maybe a bit more would be useful to describe why we need text here and not just regular HER. \n\nI think this line “triggers alearning synergy between language acquisition and policy learning” is pretty confusing and not really adding too much value. Would remove. \n\nOverall motivation makes sense, do language grounding in an interactive way much more effectively by leveraging hindsight relabeling but to get around the circular problem of hindsight generation leverage a model of successful behaviors seen thus far. This is a pretty neat thing to do!\n\nTextual HER:\n“We emphasize that such signals are inherent to the environment, and an external expert does not provide them” -> I do not think this is true. Rewards do not magically show up in the environment, they have to be provided. I get what you’re trying to say but this statement is very often not true. Please revise. \n\nConceptual question: what happens if none of the random trajectories are successful coz the reward is so sparse? Wouldn’t this be prohibitive? Importantly, I would be curious to understand how the number of entries in D affects the relabeling function m_omega and how this can be good or bad depending on the schedule of training. For instance if the D is very small at the start, it’s not going to be very good at doing the relabeling and might be erroneous.\n\nExperiments\n\n“Surprisingly, even highly noisy mappers, with a 80%noise-ratio, still provides an improvement over vanilla DQN-agents” -> do you know why?\n\nThe synthetic noisiness that you introduce is from a different distribution than the type of noisiness you’d expect from just having very few successful trajectories to train m_omega right? How can we evaluate that?\n\nIf we consider the number of successful trajectories obtained just by accident, is it even 1000 or 5000 as required to train the m_omega. If this is a negative feedback loop couldn’t it just keep getting worse because we are using erroneous m. Should we use some notion of uncertainty or something to know when to relabel with m? Or does it happen always\n\nI generally like Section 4.2 -> nicely motivated!\n\n“We emphasize again that it is impossible to have an external expert to apply HER in the general case” -> why??\n\nCan the authors introduce other baselines. For instance the recent paper from Yiding Jiang, Chelsea Finn, Shane Gu and others might be a start. Consider corpus based instruction following would be another. Maybe these can be compared in terms of the number of instructions that need to provided to it? But I think for a successful ICLR paper, we would need 1-2 more meaningful baselines. \n\n“Finally, we observe a virtuous circle that arises.” -> is there some mechanism to ensure that this is a virtuous cycle and not a vicious one? Couldn’t we just have horrible label corruption and then everything goes bad?\n\nHow easily would this scale to more temporally extended tasks in minigrid which have larger grids and more challenging tasks which are harder to solve in the sparse reward case?\n\nCan we analyze whether the language goal space has some favorable generalization properties over a state based goal space as typical HER uses?\n\nThe language analysis in Section 4.4 is quite insightful and shows the good performance of the instruction generator over time. \n\nHow would this fare as language got more ambiguous and multimodal and the instruction generator had a harder time as well as HER might generalize more poorly?\n\nRelated Work\nFu et al (From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following) might be relevant for instruction following as well, and some of Karthik Narasimhans work. \n\nLearning interactively with language would also be related to Co-Reyes et al (Guiding Policies with Language via Meta-Learning)\n\nYiding Jiang’s recent work would also be relevant (https://arxiv.org/abs/1906.07343)\n\n\nOverall I like the formulation, and it seems pretty useful for instruction following. But we need more comparisons, and a little more motivation on when this thing might become degenerate coz of the m labeling. Perhaps even a discussion/experiment on the uncertainty measure might be helpful. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers are borderline and one recommends rejection. The main criticism is the simplicity of language, scalability to a more complex problem, and questions about experiments. Due to the lack of stronger support, the paper cannot be accepted at this point. The authors are encouraged to address the reviewer's comments and resubmit to a future conference.
Decision: Reject