Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: What is the specific question/problem tackled by the paper?
This paper tackles the problem of learning language-conditioned policies from reinforcement learning. Unlike most language-conditioned navigation work which relies on human demonstrations (e.g. in the room2room environment), this work only learns from the agent’s experience using a generalization of hindsight experience replay.

Method overview:
THER (textual HER) generalizes HER to cases where goals are not in the same space as states. To deal with this gap, THER learns a mapping from state space to goal space using successful trajectories. This mapping is then used to relabel unsuccessful trajectories with a guess of what goal was reached. This intuitive approach allows the text-conditioned agent to reach  40% at a 2D navigation task when conditioned on text such as “Pick the large red circle”. 

Strengths:
The method is well motivated and would be useful. The ablations of showing how many successful trajectories are needed to learn the mapping (~1000-5000), how many time steps are needed to reach 1000 successes (~400k steps), and how accurate the mapping needs to be for HER to work (~80%) and thorough and easy to understand. This experimental completeness is itself a contribution. 

Additionally, although the authors do not discuss this, this method is actually agnostic to the particular modality (e.g. text) of the goal space and could be used anytime the goal space differs from the state space. 

Weaknesses:
The primary weakness of the paper is that the testbed environment and the textual goals are very simple. The “language” is just a list of up to 4 attributes describing the different objects and the control is simple navigation without any walls of visual variation. Additionally, the method requires accidentally getting successful trajectories early in training in order to train the mapping, and in this environment it is very easy to get successful trajectories. 
I would interested in seeing how this method would work in the room2room environment (or some other more complex task). While it is unlikely to outperform the prior methods that use the human demonstrations, it would be useful to see how close THER can get to that performance and with how many environment steps. The advantage of this environment is that it has real human knowledge, and the textual goals are limited in number, making the experiment much more realistic (as humans are unlikely to sit next to an agent and generate infinitely many diverse textual goals). 

A missing ablation in Figure 4 left is THER without waiting 400k steps before relabeling. In realistic scenarios, we would not be able to evaluate the mapper ahead of time to know when to start relabeling. How is performance affected is this knowledge is not available?

Overall, I lean to reject the paper in it’s current form, I believe this paper would be more impactful with experiments involving more language complexity or more policy complexity.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This work attempts to learn instruction following agents without the requirement for a paired instruction, behavior corpus and instead simply relying on environment interaction and using a form of hindsight relabeling to learn how to relate language and behavior. 

Introduction: 

It thus leads to the following questions: are we eventually making the rein-forcement learning problem harder, or can we generate learning synergies between policy learningand language acquisition? -> it really doesn’t seem like the point of instruction following is that. It seems like you want instruction following so that you can communicate novel objectives to your agent, with the promise of generalization. Maybe reword?

The motivation for using HER makes sense, but maybe a bit more would be useful to describe why we need text here and not just regular HER. 

I think this line “triggers alearning synergy between language acquisition and policy learning” is pretty confusing and not really adding too much value. Would remove. 

Overall motivation makes sense, do language grounding in an interactive way much more effectively by leveraging hindsight relabeling but to get around the circular problem of hindsight generation leverage a model of successful behaviors seen thus far. This is a pretty neat thing to do!

Textual HER:
“We emphasize that such signals are inherent to the environment, and an external expert does not provide them” -> I do not think this is true. Rewards do not magically show up in the environment, they have to be provided. I get what you’re trying to say but this statement is very often not true. Please revise. 

Conceptual question: what happens if none of the random trajectories are successful coz the reward is so sparse? Wouldn’t this be prohibitive? Importantly, I would be curious to understand how the number of entries in D affects the relabeling function m_omega and how this can be good or bad depending on the schedule of training. For instance if the D is very small at the start, it’s not going to be very good at doing the relabeling and might be erroneous.

Experiments

“Surprisingly, even highly noisy mappers, with a 80%noise-ratio, still provides an improvement over vanilla DQN-agents” -> do you know why?

The synthetic noisiness that you introduce is from a different distribution than the type of noisiness you’d expect from just having very few successful trajectories to train m_omega right? How can we evaluate that?

If we consider the number of successful trajectories obtained just by accident, is it even 1000 or 5000 as required to train the m_omega. If this is a negative feedback loop couldn’t it just keep getting worse because we are using erroneous m. Should we use some notion of uncertainty or something to know when to relabel with m? Or does it happen always

I generally like Section 4.2 -> nicely motivated!

“We emphasize again that it is impossible to have an external expert to apply HER in the general case” -> why??

Can the authors introduce other baselines. For instance the recent paper from Yiding Jiang, Chelsea Finn, Shane Gu and others might be a start. Consider corpus based instruction following would be another. Maybe these can be compared in terms of the number of instructions that need to provided to it? But I think for a successful ICLR paper, we would need 1-2 more meaningful baselines. 

“Finally, we observe a virtuous circle that arises.” -> is there some mechanism to ensure that this is a virtuous cycle and not a vicious one? Couldn’t we just have horrible label corruption and then everything goes bad?

How easily would this scale to more temporally extended tasks in minigrid which have larger grids and more challenging tasks which are harder to solve in the sparse reward case?

Can we analyze whether the language goal space has some favorable generalization properties over a state based goal space as typical HER uses?

The language analysis in Section 4.4 is quite insightful and shows the good performance of the instruction generator over time. 

How would this fare as language got more ambiguous and multimodal and the instruction generator had a harder time as well as HER might generalize more poorly?

Related Work
Fu et al (From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following) might be relevant for instruction following as well, and some of Karthik Narasimhans work. 

Learning interactively with language would also be related to Co-Reyes et al (Guiding Policies with Language via Meta-Learning)

Yiding Jiang’s recent work would also be relevant (https://arxiv.org/abs/1906.07343)


Overall I like the formulation, and it seems pretty useful for instruction following. But we need more comparisons, and a little more motivation on when this thing might become degenerate coz of the m labeling. Perhaps even a discussion/experiment on the uncertainty measure might be helpful. 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes THER (textual hindsight experience replay), which extends the HER algorithm to the case when goals are represented as language. Whereas in HER the mapping from states to goals is the identity function, THER trains a separate modeling network which performs a mapping from states to goals represented by language. The policy (represented as a Q-function trained via DQN) takes in the goal (a command) as an additional argument as done in HER, which allows the agent to be commanded different tasks. The authors evaluate THER on the MiniGrid environment, where they demonstrate that THER greatly outperforms vanilla goal-conditioned DQN, even in the presence of significant label noise.

Overall, combining HER with language-based goals is an interesting and novel problem, and potentially a promising approach to solving language-conditioned reinforcement learning where sparse rewards are common. The authors show fairly convincingly that THER heavily outperforms DQN, which fails to improve from the random initial policy. However, I have several conceptual concerns with the proposed algorithm:

1) There seems to be a bootstrapping problem in the algorithm, with regards to the instruction generator and the policy. If the algorithm does not succeed in reaching goals, then the instruction generator m_w has little training data. However, if m_w is not good, then the algorithm will not be able to give good reward signal to the policy. HER does not have this problem as it has an oracle goal mapping function, so m_w is always good. Evidently, the algorithm worked on the domain that it was tested in, but do the authors have any intuition on when this bootstrapping behavior could be harmful, or some justification on why it would not happen? If the language was more complex (and not limited to a small set of template instructions), would the THER approach still be reasonable?

2) How does the algorithm detect if a given goal state corresponds to successful execution of a command? Or in the notation of the paper, how is f(s,g) implemented? In general, this does not seem like a trivial question to answer if one were to implement this algorithm in a real-world scenario.

My overall decision is borderline (learning towards accept), as the experiments were well done and serve as a good proof-of-concept, but I am unsure if this approach will scale well outside of the particular tested domain.
review_title: Official Blind Review #2

Meta-review: Two reviewers are borderline and one recommends rejection. The main criticism is the simplicity of language, scalability to a more complex problem, and questions about experiments. Due to the lack of stronger support, the paper cannot be accepted at this point. The authors are encouraged to address the reviewer's comments and resubmit to a future conference.
Decision: Reject