Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work focuses on grounded reinforcement learning. Generally, it means finding a conditional RL policy that can perfectly response to high-level command. 

In this paper, the authors proposed a constraned RL algorithm RED to (1) train an agent that is capable of winning strategy games and (2) follow most human instructions. Concretely, RED uses RL to learn explorative policy while uses BC to imitate human demonstration.

Experiment results suggest RED achieves command-following policies and higher winning rate.
strengths_and_weaknesses: 
### Strengths

* The idea is concise. Use BC to learn human behavior while use RL to explore better solution.
* Visualization and videos provided are mostly clear and contain necessary information.
* Fig1 is illustrative and helpful to explain what is grounded reinforcement learning. (It would be better if Fig 1 has no background color and in PDF format)
* Code is released.

### Weaknesses

* This work assumes reward from interactive environment. (limitation)
* The code has no detailed guidance and documentation.
* Though I like Figure 2 and other experiments, the main experiment results in Table 1 is confusing and non-convincing.
* The idea is not novel. Conditional imitation learning is a long lasting topics. Refer to "End-to-end Driving via Conditional Imitation Learning"
* Many details are missing or confusing, such as Table 1 and Lagrangian method.
questions: ### Questions on method


Sec 4.3 says this work uses Lagrangian multiplier to solve the constrained optimization problem in Eq2. It turns out that, in my understanding, this is simply a weighted loss. A formal Lagrangian method should contain update process of the $\beta$, which is just ignored by this paper (or I miss, please point out). The later "iterative solution" also breaks the so called "Lagrangian method". 

---

Line 240 and Appendix A.2: Could you discuss more details on how you deal with action space? What does the "0/1 action for each unit" means? Who does the selection?

---

How to insert a NA command? IIUC, there exist a fixed set of possible commands right? So NA command should be just one more command in that set. You create embedding for each command in the set and than pass the embedding to the neural network.

---





### Questions on experiments

I can't understand table 1. IIUC, these four baselines have the policies that do not take command as input. Why they have different performance in Table 1 when applying different test commands? Please describe how you turn them as conditional policies. 

For example, since RL is trained without any objective to conduct distinguishable behaviors under different commands, why inserting different test commands will lead to different behavior?

---

I don't think Table 1 is convincing, especially when taking STD into account. 

Line 288: "Both RED and Joint policies substantially outperform Switch (BC) policy with sub-optimal commands (Random and Human Proxy), suggesting the conditioned behaviors are improved." This is a overclaimed statement. I can't draw this conclusion from Table 1.

---

I like Fig 2, which shows the dependency between commands and behaviors. Many "conditional RL" methods fail to prove that the conditional policy has casuality to the input command. This figure also shows that RED has much better "casuality" than RL.

My problem is that, in Figure 2, RL in 0% random commands achieves ~57% win rate. Does this value align with the RL performance with NA test command in Table 1?

Again, if RL is trained compelety without human command, then (1) where you can insert the command to the policy? and (2) why RL performance changes with the percentage of random commands?

---


Figure 6 says RL policy achieves similar win rate compared to RED. IIUC, RL policies are not conditioned on any human command. This result suggest that a human subject cooperates with RED trained conditional policy will lead to RL-comparable performance? This seems to be a violation to Table 1 "Oracle" result, which might lead to 

---

There is only one real gameplay footage provided (less than 2 minutes), with only 6 commands given, which is far less than convincing. Meanwhile, the commands seemed to be programmed/copy-pasted (I did not see any input lag), is there any real human intervention involved in recoding this video?

---

This game seems to be very complicated. Does your first intention of the application of this algorithm to be this high-level game? I understand the miniRTS environment is the only one that fitted the algorithm's need, but if you have alternatives (even if the performance is weak), would you also like to present them in the paper? Having diversity is a huge plus to your paper.








### Questions on paper




### Other questions or concerns



The [google drive link](https://drive.google.com/file/d/129ZV9CU6zUnp5B4v0p91Xaae8-M5zxHy/view) provided is **not anonymous**. This is a violation of the NeurIPS Code of Conduct (double-blind reviewing policy). Please action ASAP.

---

This paper has experiments involving 30 college students experiments, are attendees aware of their rights and responsibilities? A more explicit Ethics Statement should be included.

---

Conditional BC is not a new idea. Please refer to this work: End-to-end Driving via Conditional Imitation Learning

---

Appendix D, as said in the checklist, has no information on human experiment. You moved them to Appendix E.

---

The website given in the appendix is very informative and I like the real-time gameplay footage, but there are several problems I want to point out:
- Code link is broken.
- Many grammar errors, will not be present here (plural or singular, tense, etc) 

limitations: The authors provide limitation section.

GRL is an interesting field. How to improve human-AI cooperation in test-time, how to learn a better human proxy, and how to further encourage, illustrate, quantify the causity between command and behaviors might be future directions.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the problem of building reinforcement learning agents constrained by human demonstrations in the real-time strategy game MiniRTS.
strengths_and_weaknesses: **Strengths** The paper proposes an interesting approach, REinforced demonstration Distillation (RED), to find RL agents that trade off following possibly suboptimal human commands with finding an optimal solution to the MiniRTS environment. I think this is an interesting setting, and the manuscript presents the problem and related work in a clear and concise manner. The baselines the authors selected are appropriate for the problem set up

**Weaknesses**
While the manuscript claims that MiniRTS is harder than following natural language instructions in some grid world navigation environments (e.g., [1]). For starters, the problem of training from natural language seems harder for two reasons. First, understanding natural language instructions seems a harder problem than the command setup proposed in the paper. Furthermore, the problem that this previous navigation work analyzes is usually under sparse rewards (success/fail) while in the setting proposed in the paper there are dense demonstrations that may alleviate the complexity of the task.

In terms of novelty, constraining RL with demonstrations as has been extensively studied in the literature [2,3] and the architectural challenges for solving the language-grounded task were introduced in MiniRTS [4].

[1] Chevalier-Boisvert et al. BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning. 2018
[2] Yang et al. Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies. 2020
[3] Goecks et al. Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments
[4] Hu et al. Hierarchical Decision Making by Generating and Following Natural Language Instructions

questions: 1. In the human study, how did the author evaluate the level of skills of participants and how they determined when the participants were "familiar" (line 353) with the game?  
2. Could the authors clarify the difference between the framework GRL they proposed and existing language grounded approached? The manuscript would benefit from the authors addressing the difference of their setting with existing language grounded approaches. I do not believe that the complexity of MiniRTS as an environment is a sufficient argument to dismiss previous work.
limitations: The author address some limitations of their work and the setting discussed in the paper is interesting from its potential societal impact of having RL agent that trade-off following human commands with the optimality of the actions it executes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies Grounded RL, where an agent is trained to obey natural language commands while still trying to attain the highest reward. The paper proposed to iterate between a pure RL loss and a behavior cloning loss, where the BC loss takes trajectories from both human demo and from the winning ones in the current on-policy sample queue.

The method is evaluated on the MiniRTS env and dataset and it is shown that the method can balance between high winning rate when no command is given, and low winning rate when the command is random. 

strengths_and_weaknesses: Strengths:
* I appreciate the discussion around why the Lagrange multiplier is not suitable.
* The “random commands” is a good sanity check to see if the agent is affected by bad commands. 
* The assumptions and the alternatives are clearly stated (e.g. l151-152, l243-244, etc.)

Weaknesses:
* Although often observed in practice, as mentioned in l204-206, the fact that the policy can generalize at all beyond BC for any not-NA command relies on the architecture of the policy network. If that is the core, then this paper did not study the condition for which such generalization occurs. And it’s such a core assumption of the proposed method, that to leave it understudied makes me a bit uneasy.
* While I appreciate the social impact section, “should not result in any negative social impact” is a bit too extreme. One can easily imagine military uses where humans provide data in a simulator and the policy is later deployed in the real world. A stronger command-following RL agent will have a negative impact there, allowing better performance compared to an RL agent trained from scratch (which doesn’t work in the MiniRTS case). You may be able to argue that it has no worse social impact than say BC or other RL methods, but saying “any” is too strong.
questions: * Did you run any experiments to see whether the agent is able to follow certain commands with clear goals? E.g. “build two archers” is a clearly defined one and easy to evaluate if supported by the MiniRTS api. 
* How did the “switch” policy do in terms of obeying the commands, especially the not-NA commands? It would be good to compare it with that of RED, to show that both can obey commands, but RED wins more – which is a central goal you are trying to achieve, iiuc.

Comments: 
* It will be interesting to see an “adversarial oracle” where they always choose the worst dominating units to build. 
* It is interesting that “human proxy” almost always performs worse than random. 
limitations: The ratio between ||D_k|| and ||D|| and the threshold delta are dataset/env specific hparams which require extra evaluation to tune, and which will affect the command-obeying ability.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Paper introduces a new task called “Grounded Reinforcement Learning” in which agents must be reward-maximizing with respect to some high-level language “constraint” (given by another agent/human). The authors demonstrate that existing vanilla RL, constrained RL (CRL), and behavior cloning (BC) fail to perform well both with and without commands in the MiniRTS environment (an example of this problem setup).

To solve this problem setting, the authors propose a new method RED - Reinforced Demonstration Distillation, which repeatedly alternates between pure RL and BC updates.  The authors demonstrate that RED is both the strongest and most grounded policy. The authors provide, through ablations, compelling arguments. 
strengths_and_weaknesses: Strengths:

Introduces (somewhat) of a new problem statement

Clearly written 

Human Evaluation is compelling

Impactful work from a safety perspective. 

Weaknesses:

My largest criticism is the use of the word "language" and i think this undermines the entire approach - the current representation treats commands as one-hot representations. These are not learnt, shaped or any different to the state space. This has no standard components which i would consider natural language, such as pragmatics, syntax and semantics.

RED algorithm does not seem novel (effectively mixture of two)

Results are not significantly better than baselines.


questions: 1) It is unclear how this approach is different from having standard RL with a combined state space over both command and game state.  As the command representation is not learnt - it seems unclear how this actually is different to a standard RL problem.

2) Can more justification be given to how difficult the opponent you play with is? Can you play with a harder agent to more clearly demonstrate the advantages of your method?

3) It is really unclear how the command token is meaningfully different to having an additional dimension within the game state. Can some justification be given to how either this representation is learnt or has any assemblance of language (is it compositional, have semantic or pragmatic properties).

limitations: Addressing the above questions would be useful. 

I also believe actually using a language model would take great steps in demonstrating the effectiveness of your approach.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper takes on the challenge of both playing a game as optimally as possible, while following instructions and learning from demonstrations (both of which may be sub-optimal). The paper formulates this problem as a joint optimization, and tackles it by alternating steps of RL training with steps of distilling both demonstrations and successful RL trajectories. The paper evaluates this approach on the MiniRTS game, and evaluates against a range of baselines on various aspects such as rate of winning under different command types, how humans rank the policy’s instruction following, etc.
strengths_and_weaknesses: Post-response update
-----------

The authors have addressed some of my main concerns, and more thoroughly evaluated their method. I believe these new experiments could be better integrated in to the main text, and the paper is still limited by experimenting only with a single environment, but I am increasing my score in accordance with the improvements.



Original review
--------------


Strengths:
* An interesting and well-motivated problem space and proposed methods.
* In-depth experiments evaluating different perspectives on the issues raised in one domain.
* Good ablations and baseline conditions.
* Generally well written and clear. 

Weaknesses:
*The claim that the learned policy is grounded is not directly supported, only through circumstantial evidence that (while helpful) is not as direct as it could be.
  - The fact that win rates decrease with random commands is very weak evidence that the commands are being followed. The fact that humans rank the RED policy slightly higher than the rest is a little better, but the effect is weak (as noted in the paper there is an even distribution over the first rank). The statistical significance of the effect is not quantified, but there would almost certainly not be a significant difference between RED and the Switch or Joint baselines. 
  - To address this concern, it would be ideal to actually evaluate how well the agent follows commands. For example, set up a situation like Fig. 1E, give the agent a command like “Retreat”, and then evaluate whether the agent actually moves its units away or simply continues attacking, and how long it maintains this behavior. This would be far stronger, direct evidence that the agent is actually following commands (rather than e.g. doing nothing, issuing resource gathering commands without actually retreating, etc.).
  - Performing these tests in *precisely* situations where the RL policy would do something very different would be the strongest test of behavior.
* Only evaluating on a single game makes it difficult to know whether the conclusions are generalizable; it would be ideal to perform experiments in 1-2 other settings as well.
* The general approach to the prior literature could be improved. For example:
  - The paper argues that “BC and IRL algorithms assume that the demonstrations are optimal”—however, a variety of prior works have addressed this problem in various ways, both from the perspective of IRL (e.g. https://proceedings.mlr.press/v97/brown19a.html) and from BC-like approaches conditioned on rewards (https://arxiv.org/abs/1912.02875), or just by using BC to initialize RL training (e.g., https://www.nature.com/articles/nature16961). While not all these perspectives would necessarily address the issue of also following commands, they are not incompatible with it, and the paper should discuss how such approaches might relate to (or complement) the proposed technique.
  - Relatedly, prior work on agents that follow human instructions (particularly https://arxiv.org/abs/2012.05672) has relied on a combination of BC, RL, and IRL to achieve better performance than BC alone; this would be useful to discuss.
* It seems to me that there is a fundamental tension between the goals of strictly following human commands, and winning the game. The proposed approach appears to pursue a policy that effectively tries to distort all human commands towards its own objective (i.e., to follow the command in the way that is most likely to lead to winning, as in Fig. 1C vs. 1D). Phrased in this way, it seems that there are clear alignment concerns about this approach. These should be discussed as a tradeoff/risk.
  - These concerns relate to the above questions about how well the agent is following commands—if the goal is for the agent to actually strictly follow human commands, it should not e.g. follow a retreat order for one second and then go right back to attacking.
  - It seems to me that it would be *far* safer to have a system that follows human commands first and foremost—i.e., a system that is “interruptable” (https://deepmindsafetyresearch.medium.com/building-safe-artificial-intelligence-52f5f75058f1 and https://www.ijcai.org/Proceedings/2017/0032.pdf)
  - This concern is *somewhat* ameliorated by the fact that the win rates under sub-optimal commands are much lower, but it depends on how frequently the commands are issued in these conditions.
  - Regardless, these issues should be discussed.



questions: See above, but briefly:
* Could the authors demonstrate more directly how well the agents follow commands—particularly in cases where they commands are strongly divergent from the winning policy?
* What do the authors think might be the risks of the proposed algorithm?
limitations: On the whole, I think this paper could be a useful contribution. The most important changes from my perspective would be (see above for full details):
* The claim that the learned policy is grounded is not particularly well supported. Experiments on how well the agents are actually following commands would be most important for improving my rating.
* There are risks to trying to minimally follow commands while pursuing another objective. Discussing these shouldn't be a major change.

Other limitations that I don't think are as critical to address, but would be helpful:
* Only evaluating on a single game makes it difficult to know how general the advantage is.
review_title: null

Meta-review: metareview: I thank the authors for their submission and active participation in the discussions. The paper investigates natural language constrained reward maximization. While this application paper is borderline, all reviewers unanimously agree that this paper's strengths outweigh its weaknesses. In particular, reviewers noted that the paper presents an interesting [8YRG,wnXo] and impactful [WZeZ] approach, evaluated using appropriate baselines [8YRG,wnXo], with insightful qualitative analyses [oT9Z], open-source code [oT9Z] and an overall clearly written paper [8YRG,9kih,WZeZ,wnXo]. Thus, I am recommending acceptance of the paper and encourage the authors to further improve their paper based on the reviewer feedback.
confidence: Certain
Decision: Accept