Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses problem of sample efficiency and generalization in the multi-target reinforcement learning. The author proposes goal-aware cross-entropy (GACE) loss for auto-labeling goal states. The author also develops goal-discriminative attention network (GDAN) for multi-target reinforcement learning. They evaluate the proposed method on visual navigation robot arm manipulation with multi-target environment.  The proposed method in the paper compares with A3C, SAC.
main_review: - The sample efficiency and generalization for multi-goal RL are interesting problems to be addressed.
- The proposed GACE loss and GDAN method are nice contributions for research community.
- The paper evaluates good range of experiments.
- The proposed method in the paper achieves strong results against the baselines.
- The paper is well written.
limitations_and_societal_impact: - Could you please explain RS-D2 in line 18.
- The multi-target setup has also a challenge of target changing or dynamic targets, could you talk about how the method would perform in dynamic targets situations?
- In section 5, the paper talks about one of the limitation to the proposed method as target ambiguity. Could you please elaborate a little more?
- I understand that with the pandemic situation it would be difficult to access lab to conduct real-world experiments. If possible could the authors talk about how the methods can be extended to real-world robotics tasks?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an auxiliary loss function and a model architecture to facilitate RL agent’s semantic understanding. Specifically, a separate discriminator that partially shares weights with the policy is trained to select the correct goal given the input instruction, while another attention module is trained to better extract the goal-relevant features. Experiments show that the proposed method improves both the sample efficiency and generalization dramatically on instruction-following tasks in visual navigation and robot manipulation.

main_review: *Originality*: The method proposed by this paper is simple and sound. The related works are adequately cited so the contribution is clear.

*Quality*: I believe the proposed method is technically sound and used appropriately. The work is complete and both major contributions are backed up by clean ablation study. The authors are honest about their weaknesses too (e.g., it may not be well-applied to a task where the goal is not associated with a target). 

However, I wonder whether it’s true that the target also needs to be known beforehand? For example, one needs to first define a fixed set of targets that potentially can show up in the environment before training the agent. If this is true, then I think authors should add this as another weakness as it’s hard to achieve in an open-world.


*Clarity*: The paper is pretty well-written and easy to understand. The analyses in section 4.3 are very informative.

*Significance*: I think the results are cool and improves the state of the art in a demonstrable way. Additionally, other researchers will likely build on this approach to learn better representation for RL agents because it’s easy to implement.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a novel approach for improving sample efficiency in what they call "multi-target reinforcement learning". This is essentially goal-conditioned reinforcement learning where the tasks are of the form "bring me a spoon" or "go to the kitchen". Sample efficiency is improved in two ways. Firstly, they introduce a new auxiliary task of training a goal-discriminator that takes as input a state embedding and predicts which goal this state achieves. Second, they introduce a new architecture that uses attention and information from the goal discriminator to better extract task-relevant features for the policy.

Several experiments are provided to test the sample efficiency of agents trained using the method in both visual navigation tasks and robotic manipulation tasks. In both settings, their method outperforms on-policy baselines. Additional analysis is done, showing that: (1) while explicitly optimizing the state representation to improve the goal discriminator results in the fastest improvement in accuracy in the discriminator, the RL loss itself still causes the representations to contain information about the goal, just slower. This shows that the representation learning loss is helping the model learn faster what it would have to learn anyways. And finally, (2) saliency maps are provided which show that when using attention, the policy attends to more interpretable regions of the state space.
main_review: 1. Originality: While representation learning and goal-conditioned RL are popular these days, the idea of using pairs of observations and the tasks that they solve to improve learned representations and reduce sample complexity seems novel.
2. Quality: I think this is a neat idea that makes better use of the assumptions we already make when we do goal-conditioned RL (like there being some idea of "targets" that we have access to during training). I had a few questions when reading this that I am curious about:
    1. Is there a reason why on-policy RL is used rather than off-policy + HER? If a practitioner is looking for a sample-efficient algorithm for goal-conditioned RL, they will likely use some combination of off-policy RL and HER. It would be great to see whether this representation learning approach works well with those algorithms.
    2. In Section 3.2, you describe the method for collecting $s_t$ and $z$ pairs. This seems like a bit of a chicken and egg problem, where with a random policy I would imagine that there would be a very low chance of actually achieving a goal and getting a positive example, but it seems positive examples are needed to learn a good representation to learn the task. It seems like this would be trivial if you made the assumptions made in HER (that you can get a goal label for any state during training in order to relabel unsuccessful trajectories). Is this something you considered? Is there a reason you do not assume access to something like this since it seems necessary in order to evaluate the reward anyways.
3. Clarity: I think clarity is the main area where this paper is lacking. Some points:
    1. The introduction does not do a great job of motivating the approach or describing it.
    2. Section 3.3 for the most part makes sense (you are classifying states based on their achieved goal), but the motivation is not clearly described. For example, on line 137 you write "[the] vanilla base algorithm is not sufficient for gaining direct understanding of goals". What does it mean for an algorithm to be not sufficient at this task? What does gaining a "direct understanding of goals" mean? It seems like since these multi-target RL tasks are just a collection of RL tasks, that an algorithm like A3C should be able to eventually solve them?
    3. Section 3.4 is very unclear and I am still confused about GDAN after thoroughly reading the paper. I think this needs to be set up with much better motivation and described in a more intuitive way. 
    4. In Section 4.3, I am wondering how the discriminator accuracy goes up when it is not being trained anymore? This result would make sense if the discriminator was still learning but just not passing gradients back to the state encoder, but this isn't how it is described.
4. Significance: This type of representation learning approach seems applicable in many applications and seems simple to implement (at least for GACE).

Overall, I think the ideas behind the paper are solid (at least for GACE, I still need more clarity on exactly what is happening with GDAN). I would like to see the writing cleaned up and better explanations for some parts listed above.
limitations_and_societal_impact: The paper does a good job of listing some potential limitations of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method for training multi-target policies using reinforcement learning. It introduces an auxiliary classification-type loss for a feature encoder on which RL is performed, and additionally, uses an attention-based architecture to compute the policy and value function. In combination, these two elements seek to learn features which can distinguish between different goals in the environment, and then have the RL algorithm directly use these features to compute goal-relevant policy outputs. The authors show that adding these two components to existing RL algorithms improves performance and sample efficiency on simulated visual navigation and robotic manipulation environments compared to the base versions of these algorithms.
main_review: Originality: Taking features from the learned goal-discriminator and using it as an attention input is novel to me and seems quite effective. The overall method is able to demonstrate impressive performance on previously unseen visual navigation settings, likely through attending to only goal-relevant portions of the image observations. 

Quality: The method seems to provide significant improvements over the base algorithms in the experiments demonstrated. The qualitative results (Figure 5) are compelling, supporting the authors’ arguments. 

One point which could be investigated a bit more thoroughly: the GACE loss function appears to be analogous to training a classifier on goal images, which is similar to methods such as CURL but assumes that one can identify goal states for multiple tasks, which CURL does not use. If contrastive learning were performed on goal images instead, would this result in better performance than the SAC+CURL baseline presented?

Clarity:
Mostly, section 3.2 is confusing, discussing auto-labeled goal states -- how are the reached goal label z and corresponding goal state automatically collected during training? It seems the way in which the goal state is identified is through the last timestep of the episode $t^\prime$. But the termination condition seems to come from the environment, which then provides supervision. If this interpretation is correct, I feel that this is not quite completely self-supervised learning, and rather a multi-task sparse reward setting -- the environment still needs to provide the label for when the task is completed.

It would also be interesting to see how much using goal-discriminator features in the attention mechanism improves performance. What if the outputs of $\sigma$ are simply used instead?

Another useful comparison point would be the performance of training separate policies on each task (e.g., use a different actor/critic each time z changes). Because the proposed method is not goal-conditioned but rather instruction-conditioned, and so has a fixed set of possible instructions, this could be a good baseline to compare to.

Nitpick: Is the description on L138 necessarily true? This seems to depend on the particular rewards chosen for each task.

The paper is generally well-written, although as mentioned above, Section 3.2 was slightly confusing to me. 

Significance:
Using GACE+GDAN leads to vastly better sample efficiency for the navigation task, even in settings where the baseline algorithms can solve the task.

The experiments on the continuous robotic control domain are a bit weaker than the visual navigation task -- there are only three possible object positions, and the tasks are pushing and reaching, so all algorithms produce relatively strong performance. It seems that SAC+CURL is able to achieve almost the same performance (within 1 standard deviation) of GACE+GDAN, with very similar sample efficiency. Therefore it is hard to say that the method is improving much here. An even more complex manipulation task could be insightful.  

Recommendation: The ideas presented in this paper are quite nice and it also has some impressive empirical results. However, the robotic manipulation environments don't seem to significantly improve, the method could use additional comparisons as mentioned above, and it may not be suitable to tackle completely self-supervised settings as the authors claim. Therefore I feel that in its current state the paper should not yet be accepted.
limitations_and_societal_impact: As pointed out in the conclusion of the paper, there is some disparity between the performance on the two tasks, and the authors note that this is due to the degree of similarity between goal states and others (assuming it is meant the similarity between goal states and other states in the manipulation setting). I agree with this assessment. The authors also address an important limitation that tasks must specify targets, and have high reward-state correlation.
review_title: null

Meta-review: All reviewers agree that the multi-target RL settings, where the agent is to reach specified and variable goals (called targets) depending on the task specification, is an important setup and the proposed extension of RL formulation -- a loss to predict goal states and a second to focus on the features relevant for the goal -- are novel, well motivated, and empirically supported. Some of the reviewers recommend to slightly improve the writing and further clarify the self-supervision setup, which we believe can be easy accomplished for the final version. Hence accept.
Decision: Accept (Poster)