ACTRCE: Augmenting Experience via Teacher’s Advice

Abstract: Sparse reward is one of the most challenging problems in reinforcement learning (RL). Hindsight Experience Replay (HER) attempts to address this issue by converting a failure experience to a successful one by relabeling the goals. Despite its effectiveness, HER has limited applicability because it lacks a compact and universal goal representation. We present Augmenting experienCe via TeacheR's adviCE (ACTRCE), an efficient reinforcement learning technique that extends the HER framework using natural language as the goal representation. We first analyze the differences among goal representation, and show that ACTRCE can efficiently solve difficult reinforcement learning problems in challenging 3D navigation tasks, whereas HER with non-language goal representation failed to learn. We also show that with language goal representations, the agent can generalize to unseen instructions, and even generalize to instructions with unseen lexicons. We further demonstrate it is crucial to use hindsight advice to solve challenging tasks, but we also found that little amount of hindsight advice is sufficient for the learning to take off, showing the practical aspect of the method.

Reviews and author responses:


I would like to thank the authors for their detailed response and paper updates. In particular, the table illustrating the comparison with [1] is instructive and should definitely be part of the paper. My concern '2a' still stands. Focusing on the VizDoom task, it really seems to me that this paper has applied HER to the VizDoom task, and achieved greater sample efficiency, as well as models that can converge on slightly harder versions of the task. However, since the VizDoom goal is specified with language, and HER is basically agnostic about the goal specification, it is not really clear to me that ACTRCE constitutes a new algorithm in this context. As mentioned in my review, I would really encourage the authors to consider tasks which are not originally specified in language.

I will retain my rating (marginally below acceptance). 

We thank reviewer for reading our response and the revised submission. Regarding to the description about the extra requirement of the environment, we are very sorry that we mistakenly forgot to update the paper on this matter. In section 4.1, we will add another paragraph called "ACTRCE Teacher Implementation" before "Training details", that states the following:

Compared to the baseline DQN, implementing ACTRCE's teacher in our experiments required us to modify the environment to use its internal state (i.e. actual coordinates of the agent versus the objects) to generate the set of instruction goals that were reached (positive reward) or not reached (negative reward) automatically. 

We will make sure to add this paragraph in our next revised version. Thank you again for your suggestion. 


I read the authors' response and looked at the revised submission. The authors mentioned in their response that they have added the note that as compared to the baselines, the proposed method requires access to the set of goals and extra information about which goal was reached in each episode. Where was this revision added?

Paper Summary: 
The idea of the paper is to improve Hindsight Experience Replay by providing natural language instructions as intermediate goals. 

Paper Strengths:
Unfortunately, there is not many positive points about the paper except that it explores an interesting direction. 

Paper Weaknesses: 

I vote for rejection of the paper due to the following issues:

- It is not clear how a description for a point along the way is provided (when the agent is not at a target). It is not clear how those feedback sentences are generated. That is the main claim of the paper and it is not clear at all.

- The result of DQN is surprising (it is always zero). DQN is not that bad. Probably, there is a bug in the implementation. There should be comments on this in the rebuttal.

- According to several recent works, algorithms like A3C work much better than DQN. Does the proposed method provide improvements over A3C as well?

- The only measure that is reported is success rate. The episode length should be reported as well. I suggest using the SPL metric proposed by Anderson et al. in "On Evaluation of Embodied Navigation Agents".

- Replacing one word with its synonym is considered as zero-shot. That is not really a zero-shot setting. Please refer to the following paper, which is missing in the related work:
Interactive Grounded Language Acquisition and Generalization in a 2D World, ICLR 2018

- The environments are toy environments. The experiments should be carried out in more complex environments such as THOR or House3D that include more semantics.

- What is the difference between this method and providing a large negative reward at a non-target object?

- The paper discusses the advantages of word embeddings over one-hot vectors. That is obvious and not the goal of this paper. 

- It seems the same environment is used for train and test.

------------------------
Post rebuttal comments:

Most of my concerns have been addressed. My new rating is 5. I like the idea of having a compact representation for the hindsight experience replay, but there are still a few issues:

- I expected more complexity in vision and language. I do not agree with the rebuttal that AI2-THOR or House3D are not suitable. This level of complexity would be ok if this paper was among the first ones to explore this domain, but there are already several works. The zero-shot setting (changing the word with its synonym) is also so simplistic.

- The proposed method uses much more annotations than the baselines so the comparisons are not really fair. This information should have been added to the baseline to see how this additional information changes the performance. Basically, it is not clear if the improvement should be attributed to the extra annotation or the way the advice is given.

- The writing is still confusing. For instance, it is mentioned that "Concretely, for each state s ∈ S, we define T as a teacher that gives an advice T(s)", while that is not true since later it is mentioned that "the teacher give advice based solely on the terminal state". These statements are contradictory, and it is not trivial at all to provide an advice for each state.


Thank you for your suggestion. We would first want to clarify that in the ViZDoom environment, the episode will be terminated when the maximum episode length is reached, or the agent reaches (being within certain threshold distance from) an object.

We followed your recommendation and tried to come up with a metric similar to SPL. Our results indicate that, as the objects are further from the starting state (approximated by the episode length before reaching the target), the improvement of our proposed method over A3C and DQN becomes more pronounced. A3C and DQN have a drastic performance decrease for longer episodes, whereas ACTRCE remains unaffected. 

Specifically, we performed the following evaluation. We take the final trained model, and run 100 test episodes, each with different object combination and placement, noting whether each run was successful or not (binary value). We then construct a *cumulative success rate* curve, where the x-axis is the episode length, and the y-axis is the fraction of total number of episodes that were successful and had episode length *less or equal* to the x-axis value. We can see that the curve is monotonically increasing, with the y-axis being the overall success rate when the x-axis value is the maximum episode length.  This is similar in spirit to the precision-recall curve; the larger the area under this curve is, the better the model is, because it will be able to have more of successful trajectories that are short early on.

We added a new Appendix Section I to discuss this metric and show results for the 3 ViZDoom environment tasks (Figure 12). In the 5 objects hard mode (Figure 12a), we observe that all 3 training algorithms had similar performance until around episode length of 20, where ACTRCE has more successful trajectories which are longer. In the 7 objects hard mode (Figure 12b), ACTRCE maintains a similar behaviour while the baseline DQN essentially was only able to get success on very short episodes (i.e. when the target object was very close by). Lastly, in the 5 objects composition task (Figure 12c), the curve for ACTRCE indicates that there were 2 groups of trajectories: one requiring less than 10 time-steps, while another requiring over 20 time-steps. The former group occurs when the two target objects are adjacent to one another, making it easy to reach the second object after the first in only a few time steps. The latter group occurs when the target objects are not adjacent to each other, and thus requires the agent to more carefully turn around and avoid hitting other objects when trying to reach the second target.


We have used different mazes for training and test evaluation in our multi-task experiments on both KrazyGridWorld and VizDoom. Our experimental protocols and training/test splits mirror previous established works [1,2]. Our Zero-Shot Learning (ZSL) experiments are replicated from Chaplot et al. 2017 using the same train/test split for the language commands. Our results are directly comparable with the prior works on these two environments. Please see below for the details: 

For KrazyGridWorld (KGW), during training, we sample a random configuration of the maze at the beginning of the episode, as well as randomize the agent start position. This means that the positions and the colors of the goal and the lava changes when we reset the environment. We did not perform a train and test split. However, given there are many possible configurations of the maze (we have 9x9 choose 9 ~ 2.6 x 10^11 configurations if ignoring the attributes), we do not think the agent is able to memorize all mazes, and the success rates we report should be indicative of the generalization behavior.

For ViZDoom, we have a train and test split for the language commands (as done in Chaplot et al. 2017). We sample an instruction from the training set (during training) or from the test set (during the Zero Shot evaluation), and then sample the required number of objects and randomize their positions in the room. In the hard mode, the objects locations are generated via the Poisson distribution, and the agent’s position/view direction is also randomized. Therefore, each instantiation of the environment most likely has a different placement and combination of objects from any environment ever seen before. 

[1]  Chaplot et al., “Gated-Attention Architectures for Task-Oriented Language Grounding”. https://arxiv.org/abs/1706.07230. AAAI, 2018
[2] Stadie et al, “The Importance of Sampling in Meta-Reinforcement Learning”, https://arxiv.org/abs/1803.01118. NeurIPS, 2018.


Two points are missing in the rebuttal. Please address them as well:
- I had asked whether the train and test configurations are the same or not. This is left unanswered. 
- Episode length alone does not show anything. For example, an agent that learns to stop after one step has episode length of 1. Please include a 2D plot of success rate vs episode length or some metric similar to SPL that captures both success rate and episode length. 


We thank the reviewers for their response and suggestions. We have updated the paper based on their feedback, with the following changes:
 
1. Additional ViZDoom experiments with A3C reproduced baseline performance for single target with 5 objects in easy and hard mode, in Appendix G (requested by all reviewers)
2. Added average episode length in ViZDoom in Appendix H
3. Added more details on teacher advice generation in ViZDoom in Appendix B.5
4. Added more literature review, especially with works suggested by reviewer 3


Thank you for your time in reviewing our work.
 
Many of the existing works on grounded RL and language grounded navigation approaches rely on human engineered auxiliary reward [4,5] or reward shaping [6,7]. We would like to emphasize that the main focus of our work is to tackle the sparse reward, multi-goal reinforcement learning problem. In particular, we used language as the goal representation. In our experiments, we chose to use navigation-style environments in 2D (KrazyGridWorld) and 3D (ViZDoom) with goal specified with language and receiving sparse reward. We strongly disagree that these are 'toy' environments. We believe that the sparsity of the reward in the current environments we investigated, as well as their language semantics, still provide enough complexity to illustrate the benefits of our proposed framework from the perspective of reinforcement learning algorithms:
ACTRCE can solve challenging 3D navigation tasks, such as the introduced composition task with 2 target objects in ViZDoom, while using a non-language representation (i.e. one-hot representation for each instruction) failed to learn.
Little amounts of hindsight advice (1%) is sufficient for learning to take off, which can help this method be practical with real human feedback. The original HER paper always added hindsight experience during the entire training, since obtaining the goal was trivial in their case.
As with [1], our agent can generalize to unseen instructions, and in additional also to instructions with unseen lexicons during training.
 
We do appreciate Reviewer-3's feedback that there are many other indoor navigation environments (such as House3D, AI2-THOR), along with specialized metrics (such as the suggested SPL), and terminologies (such as the different types of zero shot tasks). We believe that our work can be applied to the tasks in those environments to improve the sample efficiency and performance.

We hope the reviewer can take time to revisit the revised version of our paper in light of our response, and reconsider the score.
 
We address the specific issues raised below:


>Concerns about the baselines: “The result of DQN is surprising (it is always zero). DQN is not that bad. Probably, there is a bug in the implementation.”, “Does the proposed method provide improvements over A3C as well?”
We want to emphasize that the environments we experimented on have sparse reward. DQN is known to have difficulty in performing in sparse reward environments, as noted in the original DQN paper (e.x. on Montezuma’s revenge) as well as HER paper. We have tested our DQN implementation in less challenging setting for ViZDoom, using 5 objects--instead of 7--in hard mode (single target), which is identical to Chaplot et al. 2017 set up: 

+---------------------+------------------------------------------------------------------------+-------------------------------------------------+
|                           |                           MT                                                              |                  ZSL                                        |
+---------------------+------------------------------------------------------------------------+-------------------------------------------------+
| # of frame s    |           8M          |          16M         |      150M         |  N/A  |            16M        |      150M      |  N/A |
| A3C [1]             |           -              |              -            |           -            |  0.83  |              -            |          -           | 0.73 |
| A3C (Reprod)  |   0.10 +/- 0.01 |   0.09 +/- 0.04   | 0.73 +/- 0.01 |      -    |              -            | 0.71 +/- 0.02|    -    |
| DQN                 |   0.4 +/- 0.2*   |    0.73 +/- 0.08  |           -            |      -    |   0.75 +/- 0.05  |          -            |    -    |
| ACTRCE(Ours)|*0.69 +/- 0.04*|* 0.83 +/- 0.02*|           -            |      -    |*0.77 +/- 0.02*|          -            |    -    |
+--------------------+--------------------+--------------------- +------------------+---------+--------------------+------------------+-------+

In the DQN and ACTRCE experiments, we trained for 16 million frames. In that environment set up, our DQN implementation was able to learn, but with a larger variance during training. Note that ACTRCE was able to achieve almost the same multitask (MT) performance as DQN with half the frames (ACTRCE 0.69 at 8M vs DQN 0.73 at 16M frames). As the reward becomes more sparse, then our contribution of applying hindsight feedback becomes crucial to achieving good performance in the environment. 

We had attempted to reproduce the A3C results from Chaplot et al. 2017 based on their available implementation online [2] for the single target task with 5 objects in easy and hard mode, but was not able to achieve the same published performance for the hard task, given our computation budget. Note that A3C was almost an order of magnitude less sample efficient than DQN/ACTRCE. Given the same number of frames (16M), A3C has not started to learn. Only by 150M frames that the performance of DQN and A3C was similar.

We have included this table in Appendix G in our revision of the paper. 

>“It is not clear how a description for a point along the way is provided (when the agent is not at a target). It is not clear how those feedback sentences are generated.”
We refer the reviewer to the paragraph just above section 3.1, where we state “Note that in the above formulation, we assume a MDP setting, and let the teacher give advice based solely on the terminal state”. Hence in our experiments, the teacher advice is only generated from the terminal state of the agent. We then make a copy of the transitions in the episode and replace the goal with the teacher advice and corresponding reward (1 or 0). For ViZDoom, we simply do not give any positive teacher advice when the agent did not reach any object, but we still give negative teacher advice (reward=0), describing one of the objects that the agent did not reach in the episode. We had originally tried to give the positive advice of “Reach no object” (reward = 1) when the agent did not reach any objects at the end of the episode, but it did not improve the performance. 


>“The environments are toy environments. The experiments should be carried out in more complex environments such as THOR or House3D that include more semantics.”, and “It seems the same environment is used for train and test.”

Thank you for suggesting the AI2-THOR [3] and House3D [4] environments. These environments offer more realistic 3D environment rendering than ViZDoom. However, we believe that the ViZDoom environment still provides comparable semantics as the RoomNav task introduced in House3D. Given a house, there are 5 possible rooms (ex: kitchen, dining room, etc.) and 15 possible objects (ex. shower, sofa, etc.). In ViZDoom, there is only 1 room but 18 possible objects (5 types with different colour and sizes). The RoomNav environment directly encode each goal (called ‘concepts’) as a one-hot vector, similarly to our one-hot instruction representation experiment. A particular challenging aspect of ViZDoom instruction goal is that an instruction can refer to several possible objects amount the 18 objects (ex: “Go to the blue object”), although we restrict to having only one correct object present in a particular episode. Most importantly, the current setup in RoomNav uses reward shaping via auxiliary reward based on the approximate shortest distance between the agent and the target room, and various penalties for hitting an obstacle, and time penalty. In contrast, our ViZDoom set up has sparse reward, which is aligned with the main problem of the paper. 

For AI2-THOR, the action space discretizes the scene space into a grid-world representation, due to 90 degree turn angle. In contrast, the turn angle in ViZDoom is much less than 90 degrees, leading to more fine grain control in the agent trajectory. Similarly to ViZDoom, an AI2-THOR scene is contained in 1 room, but there are 120 possible rooms belonging to one of the four room types (kitchen, living room, bedroom, bathroom). We acknowledge that the diversity of the scenes in AI2-THOR is greater. However, the arrangement of the objects in the scenes are fixed in AI2-THOR, while in ViZDoom the combination of objects present in the scene (upper bound of (18 choose 5) = 8568 and (18 choose 7) = 31824 combinations), as well as their spatial location, are randomized at each episode. This means that in the ViZDoom hard mode, every episode has most likely a unique room for navigation.

>“The paper discusses the advantages of word embeddings over one-hot vectors. That is obvious and not the goal of this paper.”

We apologize for a possible misunderstanding of the purpose of our one-hot vector instruction representation compared to the GRUs and pre-trained sentence representation. One of the purpose of the paper is to explore whether language representation of the goal can benefit HER. Then a reasonable baseline is to treat each instruction goal independently, i.e. one hot vector. Hence we approached this question by representing the goal either as a sentence (i.e. a sequence of tokens), versus as a one-hot vector (where the dimension is the number of training instructions). In both cases, the goal is eventually embedded to a fixed length vector, which is used to compute the gated attention values.  

We will make this clearer in the next revision of our submission. 

>“The episode length should be reported as well. I suggest using the SPL metric proposed by Anderson et al. in "On Evaluation of Embodied Navigation Agents".”
We agree that episode length will be an informative metric. However, it is not trivial to extend the SPL metric to our environment as the environment does not provide the optimal steps to the target. However, we will provide a plot of the average episode length over the training timesteps, which shows a decreasing trend as the training progresses. 

Please refer to Appendix H in the revised draft. 


>“Replacing one word with its synonym is considered as zero-shot. That is not really a zero-shot setting. Please refer to the following paper, which is missing in the related work:
Interactive Grounded Language Acquisition and Generalization in a 2D World, ICLR 2018”

Thank you for bringing Yu et al. 2018’s work to our attention, we will cite their paper in the related work section. In their terminology, our zero-shot results is equivalent to their ZS1 ( “interpolation” to new combinations of previously seen words for the same use case). Our one word synonym experiment was applied to the ZS1 task (i.e. the testing instructions) as well as to training instructions. We feel that this is closer to the ZS2 in Yu et al. 2018’s work, as it is extrapolating to new words transferred from other use cases and models--in this case, a pre-trained sentence embedding model. Our work differs in that we are only training the agent on the navigation task, while Yu et al. 2018 has both navigation task and a question-answering task. Their ZS2 sentences contain a word that does not appear in the Navigation task training sentence but *does* appear in their Question-Answering training answer. 

>“What is the difference between this method and providing a large negative reward at a non-target object?”

Providing large negative reward at a non-target object can lead to undesired behaviour of avoiding reaching any of the objects, only hitting walls or wandering around until the episode terminates after a maximum number of timesteps. This is because the expected return of not hitting anything is zero, compared to when the agent randomly reaches one of the (mostly likely) incorrect object and receiving a large negative reward. In comparison, our method only rewards the agent when it reaches the target object for the instruction.

References:
[1] Chaplot et al., “Gated-Attention Architectures for Task-Oriented Language Grounding”. https://arxiv.org/abs/1706.07230. ArXiv, 2017
[2] https://github.com/devendrachaplot/DeepRL-Grounding 
[3] Zhu et al, “Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning”.  https://arxiv.org/abs/1609.05143. ArXiv, 2018
[4] Yu el al., “Building Generalizable Agents with a realistic and rich 3D environment”. https://arxiv.org/abs/1801.02209. ICLR, 2018
[5] Hermann et al. “Grounded language learning in a simulated 3d world.”  https://arxiv.org/abs/1706.06551. ArXiv, 2017
[6] Misra et al.. Mapping instructions and visual observations to actions with reinforcement learning. EMNLP, 2017
[7] Das et al., “Embodied Question Answering”. https://arxiv.org/abs/1711.11543. ArXiv, 2017

Thank you for taking the time to reviewing our paper and we appreciate the positive feedback.

We will address your clarification questions below:

>“In Table 1, how many frames were DQN and ACTRCE trained for? I am wondering why the MT performance for DQN is so low. Did the DQN have Gated-Attention?”

Both DQN and ACTRCE were trained with 40 million frames, and both had identical architecture which uses Gated-Attention as in Chaplot et al. 2017 [1]. The difference with Chaplot et al’s set up is that we increased the number of objects from 5 to 7, and we also increased the size of the room by 50%. This made the reward in the environment even more sparse. We have tried experimenting on the easy and hard task using 5 objects (similar to [1]), and found that in those cases, the baseline DQN was in fact able to learn (see the next point below).

>“In Appendix D Training details,  it is mentioned that you reproduce training using Asynchronous Advantage Actor Critic (A3C), where is A3C used in the experiments?”

Thank you for reviewer pointing this out. We had mistakenly left out this result in our initial submission. We had attempted to reproduce the A3C results from Chaplot et al. 2017 [1] based on their available implementation online [2] for the single target task with 5 objects, but was not able to achieve the same performance in the hard mode as published, given our computation budget. 

+---------------------+------------------------------------------------------------------------+-------------------------------------------------+
|                           |                           MT                                                              |                  ZSL                                        |
+---------------------+------------------------------------------------------------------------+-------------------------------------------------+
| # of frame s    |           8M          |          16M         |      150M         |  N/A  |            16M        |      150M      |  N/A |
| A3C [1]             |           -              |              -            |           -            |  0.83  |              -            |          -           | 0.73 |
| A3C (Reprod)  |   0.10 +/- 0.01 |   0.09 +/- 0.04   | 0.73 +/- 0.01 |      -    |              -            | 0.71 +/- 0.02|    -    |
| DQN                 |   0.4 +/- 0.2*   |    0.73 +/- 0.08  |           -            |      -    |   0.75 +/- 0.05  |          -            |    -    |
| ACTRCE(Ours)|*0.69 +/- 0.04*|* 0.83 +/- 0.02*|           -            |      -    |*0.77 +/- 0.02*|          -            |    -    |
+--------------------+--------------------+--------------------- +------------------+---------+--------------------+------------------+-------+

In the DQN and ACTRCE experiments, we trained for 16 million frames. We will include these results for the 5 objects (easy and hard mode) in the next revision in Appendix G. 

>“The composition task is very interesting, did the agent receive intermediate rewards for completing a part of the instruction in this task?”

In the composition task, the reward is still sparse--the agent only receives a reward of 1 if the agent reached both of the desired objects during the episode, in any order. A reward of 0 is given otherwise, i.e. when the agent reached only none/one object, or two objects that are not both desired ones. 

>“In Appendix D Training details, what do you mean by 'chosen from the range {1000, 10000, 10000}'?”

The third number was a typo (100000). We simply wanted to denote that we tried those 3 replay buffer sizes when performing hyperparameter tuning. 

>“It is important to note that as compared to the baselines, the proposed method requires access to the set of goals and extra information about which goal was reached in each episode.”

We will note this in our revision of the paper!

[1] Chaplot et al., “Gated-Attention Architectures for Task-Oriented Language Grounding”. https://arxiv.org/abs/1706.07230
[2] https://github.com/devendrachaplot/DeepRL-Grounding 

Thank you for your time reviewing our paper and providing constructive feedback. The two main concerns are (1) the lack of comparison to existing work and (2) evaluating on tasks that already uses language specification. We provide experimental results for (1) and make additional justification for (2).

Concern 1: >”It would seem like a natural comparison would be to take the model from Chaplot (leaving the task and architecture etc unchanged) and train it using ACTRCE.”

We indeed took the architecture from Chaplot et al. 2017 and removed one of the head, to predict only the Q-value instead of actor policy and critic heads for the A3C, for our DQN/ACTRCE training. For the 5 objects in hard mode (single target) we achieved similar performance to Chaplot et al. using ACTRCE, but with *an order of magnitude more sample efficiency* (16 million frames vs 150 million frames): 

+---------------------+------------------------------------------------------------------------+-------------------------------------------------+
|                           |                           MT                                                              |                  ZSL                                        |
+---------------------+------------------------------------------------------------------------+-------------------------------------------------+
| # of frame s    |           8M          |          16M         |      150M         |  N/A  |            16M        |      150M      |  N/A |
| A3C [1]             |           -              |              -            |           -            |  0.83  |              -            |          -           | 0.73 |
| A3C (Reprod)  |   0.10 +/- 0.01 |   0.09 +/- 0.04   | 0.73 +/- 0.01 |      -    |              -            | 0.71 +/- 0.02|    -    |
| DQN                 |   0.4 +/- 0.2*   |    0.73 +/- 0.08  |           -            |      -    |   0.75 +/- 0.05  |          -            |    -    |
| ACTRCE(Ours)|*0.69 +/- 0.04*|* 0.83 +/- 0.02*|           -            |      -    |*0.77 +/- 0.02*|          -            |    -    |
+--------------------+--------------------+--------------------- +------------------+---------+--------------------+------------------+-------+

Performance averaged over 2 random seeds. We used the available online implementation for A3C [2] for reproducing the results, but was unable to match the published performance.
We will include these results (including training plots) for the 5 objects (easy and hard mode) ViZDoom environment in the next revision in Appendix G.

Concern 2a: >“In the VizDoom task, the goal specification is already in (templated) language. Given that this is the case, and the mapping from states to goals can be extracted from the environment anyway, it seems like the method that is applied really just reduces to a vanilla implementation of HER.”

In general, the HER framework requires a strategy for sampling goals for replay and the reward function relating the state, action, and goal to a scalar reward. In this sense, our work is one instantiation of HER, where the sampling strategy are the teachers’ language advice (for achieved goals with reward = 1 or unachieved goals with reward = 0). We are investigating a non-linear mapping between the state and the goal space (i.e. from pixels to sequence of tokens), in contrast to the linear mapping of the vanilla HER where the goal space is the same as the state space, or a subset of the dimensions of the state space. 

Another distinction with the original HER is that they assumed that a state only satisfies one goal. However, in our case, the mapping from state to goal is one-to-many: there can be many different language sentences that describe what the agent accomplished. An agent reaching a short blue torch can be described with “Go to short blue torch”, or “Go to torch” (if there are no other torches in the environment), or “Go to the blue object”, etc. In the original HER, to generate goals implies sampling from the states visited, usually in the episode. With our set up, the goals can be from the set of training instructions that the agent has not encountered before as well. 

[1] Chaplot et al., “Gated-Attention Architectures for Task-Oriented Language Grounding”. https://arxiv.org/abs/1706.07230
[2] https://github.com/devendrachaplot/DeepRL-Grounding 


Concern 2b: > “I expected the ACTRCE approach to be applied to a task where the goal was not originally specified in language, perhaps by collecting language from human teachers. This would be a much more interesting experiment, addressing the question of whether human feedback in natural language can help the agent learn more quickly.”

We agree that applying the ACTRCE to a problem which the goal was not originally specified in language by collecting language from human teachers would be an interesting approach.  In our work, we took a different approach where we converted a task originally specified in language and remove the language aspect, turning an instruction into one hot vector. In this setting, we were able to show that as we transitioned from single to composition task, using language representation helped the agent learn better than using a one-hot approach for each instruction

Smaller concern: >"ACTRCE - possibly the most tortured acronym in recent memory! How should it be pronounced?"

ACTRCE is pronounced as "actress" 

This paper considers the assumption implicit in hindsight experience replay (HER), namely that we have access to a mapping from states to goals. Rather than satisfying this requirement by defining goals as states, which involves great redundancy, the paper proposes a natural language goal representation. Concretely, for every state a teacher is used to provide a natural language description of the goal achieved in that state, which can be used to directly relabel the goal so the episode can be used as a positive experience.   

Strengths:
- the proposed idea is simple and intuitively appealing, and shows much better results than the DQN baseline.

Weaknesses:
- In the VizDoom task, the goal specification is already in (templated) language. Given that this is the case, and the mapping from states to goals can be extracted from the environment anyway, it seems like the method that is applied really just reduces to a vanilla implementation of HER. There seems to be little novelty in this. From reading the introduction and method, I expected the ACTRCE approach to be applied to a task where the goal was not originally specified in language, perhaps by collecting language from human teachers. This would be a much more interesting experiment, addressing the question of whether human feedback in natural language can help the agent learn more quickly.
- Even leaving aside the previous concern, it seems very difficult to put this work in the context of previous work on the same tasks. For example, it is not clear why there are no comparisons to the previous work on instruction following in VizDoom, as the setting appears to be exactly like Chaplot et al. 2017. It would seem like a natural comparison would be to take the model from Chaplot (leaving the task and architecture etc unchanged) and train it using ACTRCE. Is there any reason why this can’t be done? There is already so much existing work in this space, it seems quite unusual that the proposed new method is not compared to any existing work on an existing task.


Summary:
This is a simple and intuitively appealing idea, but I find the evaluation to be quite lacking because the tasks already use a language specification (such that ACTRCE seems to be vanilla HER in application) and there are no comparisons to previous work. These two concerns seem quite substantial to me and make it difficult to recommend acceptance. 

Smaller issues:
- ACTRCE - possibly the most tortured acronym in recent memory! How should it be pronounced?

This submission presents a method to improve the sample-efficiency of instruction-following models by leveraging the Hindsight Experience Replay framework with natural language goals. 

Here are  my comments/questions:
- The paper is well written and easy to follow, it introduces a simple idea which achieves very good results.
- In addition to improving the performance as compared to the baselines, the authors perform a wide variety of experiments such as analysis of language representations, visualization of embeddings, etc. which lead several insightful results such as ability of sentence embeddings to generalize to unseen lexicon, ability of the model to perform well with just 1% advice.
- It is important to note that as compared to the baselines, the proposed method requires access to the set of goals and extra information about which goal was reached in each episode.
- In Table 1, how many frames were DQN and ACTRCE trained for? I am wondering why the MT performance for DQN is so low. Did the DQN have Gated-Attention?
- The composition task is very interesting, did the agent receive intermediate rewards for completing a part of the instruction in this task?
- Some implementation details questions:
	- In Appendix D Training details, what do you mean by 'chosen from the range {1000, 10000, 10000}'?
	- In Appendix D Training details,  it is mentioned that you reproduce training using Asynchronous Advantage Actor Critic (A3C), where is A3C used in the experiments?

Meta-review: This paper was reviewed by three experts (I assure the authors R3 is indeed familiar with RL and this area). Initially, the reviews were mixed with several concerns raised. After the author response, R2 and R3 recommend rejecting the paper, and R1 is unwilling to defend/champion/support it (not visible to the authors). The AC agrees with the concerns raised (in particular by R2) and finds no basis for overruling this recommendation. We encourage the authors to incorporate reviewer feedback and submit a stronger manuscript at a future venue. 

Decision: rejected-papers