Grounded Reinforcement Learning: Learning to Win the Game under Human Commands
Authors: Shusheng Xu
Huaijie Wang
Yi Wu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Language Grounding
Human-AI Interaction

Abstract: We consider the problem of building a reinforcement learning (RL) agent that can both accomplish non-trivial tasks, like winning a real-time strategy game, and strictly follow high-level language commands from humans, like “attack”, even if a command is sub-optimal. We call this novel yet important problem, Grounded Reinforcement Learning (GRL). Compared with other language grounding tasks, GRL is particularly non-trivial and cannot be simply solved by pure RL or behavior cloning (BC). From the RL perspective, it is extremely challenging to derive a precise reward function for human preferences since the commands are abstract and the valid behaviors are highly complicated and multi-modal. From the BC perspective, it is impossible to obtain perfect demonstrations since human strategies in complex games are typically sub-optimal. We tackle GRL via a simple, tractable, and practical constrained RL objective and develop an iterative RL algorithm, REinforced demonstration Distillation (RED), to obtain a strong GRL policy. We evaluate the policies derived by RED, BC and pure RL methods on a simplified real-time strategy game, MiniRTS. Experiment results and human studies show that the RED policy is able to consistently follow human commands and achieve a higher win rate than the baselines. We release our code and present more examples at https://sites.google.com/view/grounded-rl.

Grounded Reinforcement Learning: Learning to Win the Game under Human Commands
Shusheng Xu', Huaijie Wang's and Yi Wu III. Tsinghua University, Beijing, China 2 Shanghai Qi Zhi Institute, Shanghai, China {xuss20, wanghuai19] @ mmails.singhuu..ddu.cr
jxwuyi@ gmail.com
Abstract
We consider the problem of building a reinforcement learning (RL) agent that can both accomplish non-trivial tasks, like winning a real-time strategy game, and strictly follow high-level language commands from humans, like "attack", even if a command is sub-optimal. We call this novel yet important problem, Grounded Reinforcement Learning (GRL). Compared with other language grounding tasks, GRL is particularly non-trivial and cannot be simply solved by pure RI. or behavior cloning (BC). From the RL perspective, it is extremely challenging to derive a precise reward function for human preferences since the commands are abstract and the valid behaviors are highly complicated and multi-modal. From the BC perspective, is impossible to obtain perfect demonstrations since human strategies in complex games typically sub-optimal. We tackle GRL via simple, tractable, and practical constrained RL objective and develop an iterative RL algorithm, REinforced demonstration Distillation (RED), to obtain strong GRL policy. We evaluate the policies derived by RED, BC and pure RI. methods on a simplified real-time strategy game, MiniRTS. Experiment results and human studies show that the RED policy is able to consistently follow human commands and, at the same time, achieve higher win rate than the baselines. We release our code and present more examples at httttt///ssteeee /ttt::://iiteegoogglee com cwwcccooiiiiieiee view/
wwwo///iwww/rouneeeeeeeee grounded-rr
Introduction
Building assistive agents that can help humans accomplish complex tasks remains a long-standing challenge in artificial intelligence. Natural language, as the most generic protocol for humans to exchange and share knowledge, becomes a general and important interface for human-Al interaction. Decades of research efforts have been continuously made to develop Al systems that can ground agent behaviors to natural language commands [69 [64 46
Recently, as deep reinforcement learning (RL) techniques have been widely used to develop interactive agents solve a variety of challenging problems, it becomes new trend cast language grounding as an RL problem to train agents that can automatically ground language concepts to visual objects or behaviors an interactive fashion 22 13 [5]. In particular, an RL agent is typically presented with language command describing the goal of the RL task and will receive a success reward when the desired goal state under the command is achieved. A well-trained RL agent can often exhibit strong generalization capabilities to novel commands. Despite the simplicity and effectiveness, such an RL formulation requires a language generator to repeatedly output random commands and an explicit reward function to determine whether the command is accomplished. Hence, most cxisting RL works focus on simple navigation problems with template- based compositional languages over objects and attributes, e.g., "go to the red box next to the blue wall" [5 68 34 15
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A
MiniRTS Environment
Observe
Our Unit
Enemy Unit
the
Enemy Base
Datermina the action type (e.g. attack, move, build)
Resource
of
Choose the target
River
Enw stop
Gross
RL: No commands
Human playing:
GRL: The agent actions
GRL: The agent strictly follows the command
bahaviors are sub-optimal
Altock Torget
Altock Target
Retreat even
Dying
Atlack
Attack
Retreal
O wall-trained agent
commander
commander
commander
well-trained
agent
Reinforcement Learning
Human Demonstration
Ground Reinforcement
Ground Reinforcement
Learning
Learning
Figure The grounded reinforcement learning (GRL) problem on the MiniRTS environment. (A) MiniRTS is a real-time strategy game where the player in blue needs to control its units to kill the enemy units red. (B) conventional RL agent. (C) MiniRTS provides a dataset of human demonstrations in the form of paired abstract language commands (e.g., "attack") and control action sequences. Human actions are often sub-optimal. (D) GRL aims to learn a command-conditionn agent such that it plays winning strategy stronger than the human executor. (E) A GRL agent should strictly follow the human command even if is sub-optimal.
To train agents that can interpret general human languages, another research direction is to leverage paired behavior-commant data collected from human demonstrators. Representative applications include robotic manipulation, where the robot needs to map language commands to predefined motion skills 160 35 61 65), and vision-and-latt lad--uugee navigation, where an agent learns from human demonstrations to navigate towards desired location in a 3D environment [51] S1IL 13 29 71]..
Since an accurate command-followin reward no longer accessible, most works apply behavior cloning (BC) to directly imitate human behaviors or adopt inverse reinforcement learning (IRL) to 17.. Although the use of human methods require massive demonstrations,
learn language-condie conditioned reward function 19 52
data enables natural commands, pure imitation-basee which can be expensive collect. More importantly, both BC and IRL algorithms assume that the
79 17), in the sense that (i) each behavior demonstration optimal under its paired language command, and (ii) the language commands are optimal for the underlying task. Such an assumption is reasonable in restricted domains like manipulation or navigation, where the optimal behavior simply the shortest trajectory to the goal while human commands typically provide strong guidance towards task completion. However, in more complex problems like playing real-time strategy games, it is often the case that human players can be highly biased or sub-optimal [10].
demonstrations are optimal [19 22 30
23
Let's consider concrete example simplified real-time strategy game, MiniRTS 33
Fig. []]
shows scenario where human commander gives an abstract command "attack" to human executor to attack enemy units. However, the action taken by the human executor sub-optimal: a unit with full HP is under attack while clear better strategy is to attack another dying unit. Moreover, the human commander can be sub-optimal as well. As shown in Fig. [T), very few enemy units are remaining, so the optimal command should be simply "attack" for the win. However, the commander sends

---

"retreat" command. This could be possibly due to the partially observed game state or human biases, but an obedient executor should still faithfully control the units to stop attacking and retreat.
We study this novel RI. challenge, i.e., learning an strong agent capable of not only achieving a high win rate in real-time strategy game, e-g., MiniRTS, but also strictly following high-level language commands, even if a sub-optimal command is given. We call this problem. Grounded Reinforcement Learning (GRL), which assumes an interactive RL environment with a dataset of (sub-optimal) human demonstrations reflecting proper human behaviors under language commands. In this setting, the rewards are not associated with the language commands, and is non-triviaa conttrivial to verify whether high-level language command is completed or not. So the policy can only learn about how humans follow commands from the demonstrations. We remark that GRL is different from standard language grounding problems since the primary mission is not towards better language understanding via interactions [TI]. Instead, GRL. focuses on the opposite direction, i.e., learning strong winning strategies while taking human commands as high-level behavior regulations, which is more related to the concept of developing compatleee Al [57]
To tackle the GRL problem, we propose simple yet effective constrained RL objective as a tractable approximation and developed an iterative RL algorithm, REinforced demonstration Distillation (RED), to derive a strong language-conditioned policy. RED adopts unconditioned RL training to encourage the policy to explore stronger winning strategies and periodically applies self-distillation and BC over demonstrations to ensure that the policy behavior is consistent with human preferences. We evaluate the performance of RED as well as baseline methods including BC-based methods and RL variants on the MiniRTS environment. Simulation results show that RED policy achieves strong command-followingg
capabilities and higher win rate than baselines under various types of test-time commands. We also conduct human evaluation by inviting 30 volunteers to play with policies trained by different methods. RED policy appears to be the most obedient under human votes and leads to a much higher collaborative win rate with human commanders.
2 Related Work
Language grounding. The idea of grounding languages to behaviors or concepts can be traced backed to 1970s (69]. and most early works assume simple domain-specifte languages and a pre- defined set of skills [45] 777 37 5). Thanks to the recent advances of deep RL, it becomes feasible ground languages to visual concepts and non-trivial behaviors in simulated world. Many 3D environments with
goals have been developed over different domains, includ-
ing maze exploration [15). visual navigation [33 29 71 and robot control B5 61 65), allowing end-to-end concept learning and grounding. RL training requires a goal generator and a reward function. Hence, these testbeds only adopt template- based languages over objects (e.g.. box eube) and attributes (e.g., spatial relation or color) and assume an oracle that can verify whether a state is consistent with the language description. We focus following high-level natural commands.
Regarding natural commands, recent visual navigation benchmarks 3 49 61 62 started to provide large-scale human demonstrations with the goal or pathway specified by natural languages. Behavior cloning (BC) is feasible since every human description is paired with a successful trajectory [19 52 30.. RL fine-tuning can be also applied, since a precise success measurement (i.e., distance to the goal position) w.r.t. each language instruction is known [2 67), Some works adopt inverse RL (IRL) to learn a lunguage-conditions are actually optimal. We consider a more challenging game MiniRTS 333 with abstract language commands and complex behavior space. BC and IRI. methods work poorly in MiniRTS since both language commands and game trajectories from human annotators are highly sub-optimal.
reward function 23 79 thanks to the fact that the demonstrations
There are also successful attempts to leverage powerful pretrained language models to map general languages to the goal space [36 4 or a predefined set of skills [3 60 2 so that massive paired demonstrations can be no longer necessary. We primarily consider the problem of policy learning, so the use of pretrained model is orthogonal to our current focus but remains an exciting future direction.
There are also parallel works on language grounding in other domains, such as text games [38], answer questioning 4 and communication [39], where languages serve as state descriptor or action space rather than commands to follow. Our work is also related to ad-hoc team play [66 12 since
the policy cooperates with arbitrary commands. The difference is that the commander is not assumed to be optimal under the game reward. The policy must be assistive (28) 70 or even obedient [50].
3

---

Constrained reinforcement learning. We cast grounded RL as a constrained RL (CRL) formu- lation for tractability. CRL algorithmss often leverage the Lagrangian multiplier [55] or projected gradients [1] [2] for policy improvement assuming simple (closed-form) constraints over states. In our setting, constraints can be only implicitly learned from human demonstrations. We simply adopt the behavior cloning objective, i.e., negative log likelihood, as the constraint satisfying metric, which is related to RL with demonstrations (RLwD) [32 25]. However, due to limited data and a significant distribution shift between demonstrations and on-policy trajectories, RLwD methods can perform poorly. In addition, [73] iteratively optimizes the RL objective, projects the policy on a region around reference policy, and enforces the policy satisfies the cost constraint. (27] updates the policies on the mixture of the expert replay buffer and the rollout trajectories. The most related work to our RED algorithm is Supervised Seeded Iterated Learning (SSIL) 42], which is designed for learning grounded communication and adopts a conceptually similar iterative framework by alternating between RL and self-distillation. SSIL suggests jointly optimize RI. and BC objectives assuming human communications are perfect demonstrations. By contrast, we find that an additional BC loss can be harmful to RL training due to the distribution shift issue. Similar empirical phenomena have been also reported in multi-task RL literature 75 63 74
3
Preliminary
3.1 The MiniRTS Environment and Dataset
is a grid-world RI. environment (Fig. D that distills the key features of complex real-time strategy games. It has two parties, a player (blue) controlled by human/policy against built-in script AI (red). The player controls units to collect resources, do construction and kill all the enemy units or destroy the enemy base to win game. A total of unit types form rock-paper- scissors attacking dynamics. Learning a strong policy is challenging due to partially observed states, complicated micro- managemmnn over dozens of units, and extremely diverse possible strategies.
MiniRTS 33
In addition to an RL environment, MiniRTS provides a fommand--ollowing dataset, which is collected by two humans playing collaboratively against the script opponent. One is the commander, who gives high-level strategic language commands like "attack", "retreat", or "build an archer". The other one is the executor, who controls the units to follow the language command. We remark that human data were collected against much weaker scripts than the default game opponent. Even in such an easier mode, the collected game episodes are highly sub-optimal with a win rate of merely 47.2%. The dataset contains a total of 63,285 commands with an average of 13.7 per game while the average game horizon is 151.7 So, each single command may take a long horizon to accomplish. For notation simplicity in this paper, we assume an independent command is given to the executor at each time step, although a command should be repeated for a few steps. Full details are in appendix.
3.2 Notation
Reinforcement learning. We formulate the MiniRTS environment as Partially Observable Markov Decision Process (POMDP), which is represented by tuple (8, A, O,r,p). S is the state space. A is the action space. O is the observation space. r r: S x A + IR is the reward function. p S A x S + [0, 1] is the transition probability with p(6'1s, a) denoting the probability from state s to state s' after taking action At each time step. the agent (i.e., the game player) observes or = (s:). produces an action a according to its policy me parameterized by 6, and receives a reward r, = r(st, a.). The optimal policy should achieve high win rates in the game, namely 0* arg max, E,,.., [E, r(s,,, a.)]. Note that we omit the discount factor y for conciseness only.
Command-following policy, We represent the policy by (a lo:, ct). which conditions on an observation o, E O and a language command e, C at each time step t. c, can be generated from arbitrary distributions, C' denotes the space of possible natural language commands. An LSTM is used encode a command e an embedding. We use a special command "NA" denote "no command". which corresponds to a zero embedding. When setting every G to be NA, the command-follovin policy अe degenerates a conventional policy in classical RL without language commands.
'Our numbers slightly different from 33 because the original data- cata-processing script is not released.

---

Human demonstrations. We denote human demonstrations as dataset of trajectories, i.e., D = (Ti: T2...... ...). Each trajectory 7;. denoted by тi (s), ag c. si. aj.ci. .i....), consists of transitions from single game played by human executor and the paired command sequence from a human commander. Human behaviors can be sub-optimal: i.e., given the observation of O(s)) and the command e the chosen action of can result in low rewards. Likewise, a command eg can be arbitrary w.r.t. human preferences. Finally, since human commanders are always asked to generate language commands, D does not contain any NA command, i.e., ej o NA.
4 Method
4.1
Grounded Reinforcement Learning: Problem Formulation
The mission of GRL is to learn a policy To such that can achieve high rewards in the envi- romment and follow any possible command E by achieving consistent behaviors with human demonstrations D. We formulate the GRL problem as the following RL objective Jg:
(6)
E
o2 GCEC
Er(ss ai)
subject K(TO, D) S 5,
de (c),
where K denotes distance metric between the policy behaviors from my and the human data DD The commands can arbitrary. To needs follow any possible command c E while still attains the highest rewards under the behavior constraint K (TT,, D) S 8.
Remark #1:
Remark #2: Since the human actions in D can be sub-optimal w.r.t. the paired commands, can be problematic to directly perform behavior cloning (BC) over D. Therefore, we utilize a threshold 8 so that the command-followw following policy Ty can possibly deviate from the sub-optimal behaviors in D.
Note that Eq. iss generally intractable and is presented only for the purpose of problem definition.
4.2 Constrained RL as a Tractable Approximation for GRL
There are two issues when directly optimizing Eq. ( First, the command es can be arbitrary. Second, the choice of behavior distance metric K ((T,, D) should be specified.
Training commands. An obvious choice is sample random command o from C at each time step t. which is perfectly aligned with Eq. (... However, inconsistent commands from random sampling are typically harmful for win, making the RI. agent tend to ignore the commands. An alternative is "human proxy" by learning command model he.(cl)s)) over human data 'D, leading to the objective (e (..raooo [E, r(a,, [[)] Although this is reasonable, we need to be aware that human commands are sub-optimal and biased, which may further limit the chance for the policy me to discover strong strategies during RL training. Thus, we propose an extreme version: i.e., directly performing unconditioned RL training by setting every command c, to be NA. This allows the policy to freely explore the strategy space during RL training for the best winning strategies.
Command followwng metric. The simplest distance metric K is the behavior cloning (BC) loss, i.c., negative likelihodo (NLL) of TTg for (st. ae, ot) D. Specifically, we can define BC objective (()) IP |- log To (o(a (az|or, cx)] and constrain (() < 5 to ensure the policy behavior does not deviate from the demonstrations too much. There can be alternatives, such as KL--divergence or learned metric. Regarding KL-divergence, since we are measuring the distance between samples D and distribution 7T. LL((D) (T) is equivalent to L(0) while K L requires an behavior proxy over D, which may involve additional biases. Regarding a learned metric, we can apply inverse RL to derive reward function on whether the command is accomplished (23 G. Z]. However, we empirically notice that learning- based metric works poorly on MiniRTS. We hypothesis that this is because the demonstrations are sub-optimal and possibly limited in size for such a complex game.
Tractable objective. To sum up, we propose the following constrained RL objective Jc(0) as tractable objective for the GRL problem, i.c..
(0) = Eny~sry
Σ r
subject to L(0:D) = Ep log (o 5 (2)
5

---

Note that in Eq. ( the sample distribution from the RL objective Je is disjoint from the BC objective (0: D) since we force at NA in RL training while D does not contain NA all. Hence, Eq. (2) may look a bit ill-posed: an obvious solution is a "switching" policy which takes either the pure RI. strategy or the BC strategy with or without a command. However, we will empirically show that such switching policy does not provide strong performances. The insight is that we are training a conditioned neural policy. It is often observed in practice (e.g., multi-task learning) that neural network can implicitly distill strategies from different modalities via its shared representations [20] 14 76). Therefore, the winning strategy discovered from RL training can empirically improve the sub-optimal command following demonstrations within bounded range. This phenomenon can be also justified by some recent theoretical findings [48] 16.
command-following
4.3 Reinforced Demonstration Distillation
straightforward way to solve the constrained optimization problem in Eq. Lagrangian multiplier, which leads to a joint optimization problem, i.e.,
2 is to adopt the
(0) = Jc(0)- - 3L(0; D).
Eq. 3 treats the BC objective L(0; D) as an auxiliary loss of the stardard RL objective Jc(0). The constraint in Eq. (2) can be satisfied by tuning the coefficient 3. However, due to the distribution shift issue between on- on-policy samples and demonstrations, the supervised learning loss may interfere with the policy gradient and further makes the training process sensitive and unstable [21 40).
Iterative solution. Inspired by the recent advances self-imitation [53] and self-distillation 77 43 42), we adopt an iterative framework to decouple the RL loss and the BC loss in Eq. (3). The idea simple: for RI. training, we solely estimate the unconstrained policy gradient over Jc without considering the BC loss; after policy improvement, we distill the demonstrations into the policy by running pure behavior cloning over both human demonstrations and self- ceff-generated winning trajectories. By repeatedly alternating between pure RL and pure BC, are able to achieve stable learning process. We call this method, REinforced demonstration Distillation (RED).
Specifically, let 07 denote the parameters at iteration k. then we have the following update rule.
RL phase:
@RL f- 0 + (Okk(()) 0 00L (OR DU DA), Dz = ( (ii()s, ~ Tep (,NA)).
(4)
BC phase:
(5)
Here a denotes the learning rate. By repeating the update rules for N iterations, we will derive our final parameter f* The remaining issue is to ensure the constraint is satisfied, i.e., T,;0; D) S 6. We empirically notice that this can be accomplished by controlling the ratio between the size of on-policy samples [D& || and the size of demonstrations |
We present an intuitive justification of the self-distillation process from the perspective of sample distributions 9 g We adopt RL for the highest rewards, so the unconditioned policy may frequently visit those states approaching a win. As result, for games played by strong humans with aggressive commands, the paired game states will be more aligned with the RL state distribution while the states produced by biased or defensive human commanders will be more apart. As we are distilling two drift distributions into single neural policy To, it will be more likely to get actions over those overlapping states improved within a bounded BC loss during optimization (see evidences in Sec. 5.3)
4.4 Implementations
We highlight some critical implementation factors here. More details are in appendix.
Policy architecture. We adopt a similar policy architecture to the provided executor model in MiniRTS (33 but with modified action space and an additional value head as the critic. The original model outputs an action for every controllable unit, which makes RL training extremely slow. We introduce additional 0/1 selection action each unit denoting whether this unit should act not, which substantially reduces the action dimension.
Policy learning. We use PPO 59 for RL training, which involves multiple mini-batch policy gradient steps. Since RL. from scratch fails completely in practice, we fine-tune the BC policy and pretrain the value head with the policy parameters frozen. We also empirically find learning rate

---

particularly helpful for stabilizing training with a pretrained model: we start with learning rate and then linearly increase the desired value a.
warmup A1
Creating Dr. Note that in the early stage of training phase, the winning rate can be low. In order to obtain sufficient data for Dr. we implement queue to store all past winning trajectories and use bootstrapped sampling if the queue is still not full. For the ratio between ||D and |[D|| we ensure the BC loss (i.e., negative log likelihood) of ι over the validation set is at most 10% more than the pure BC policy (i.e., no RL training). We can also run sub-sampling over 'D to reduce the required sample size. Empirically, we find that simply setting ||Dx|l ||]]] leads strong performances.
5 Experiment
All the simulation win rates are based on 1200 test games and repeated over 3 random seeds. More details and additional results including emergent behaviors are deferred to appendix.
Baselines. We consider the following methods addition to our RED algorithm as baselines.
1. "RL" first pretrains the policy over demonstrations and then performs pure unconditioned RL training (i.e., conditioning on "NA" command) without any command-follon following constraint.
2. "Switch" is the "switching" policy, which consists of two policies, a pure BC policy from demon- strations and pure RL policy without commands. When langauge command given, it runs the BC policy and uses the RL policy otherwise.
" "Joint" optimizes the soft objective Je in Eq. (3). B is selected via grid-search process.
4. "IRL" learns a reward function from demonstrations (4 and combines the game reward and the learned language reward to train conditioned policy.
Evaluation. To evaluate a strong GRL policy, we need to answer the following questions.
1. Does the policy learn a strong winning strategy? The win rate can be direct metric for this question. A RED policy should largely improves a pure BC policy with a high win rate.
2. Does the policy follow the commands well? A good GRL policy should keep its winning strategy while follow arbitrary commands. A precise evaluation of this question is non-trivial since the constraint satisfying condition, i.e., BC loss, can be very misleading signa(" ?
We adopt an approximate measurement via win rates in Sec. Bsc.B.Iaam and then conduct human evaluation in Sec. 5.4
We also conduct study on OOD commands and algorithmic hyper-parameter in Sec. 5.2 and Sec. 5.3
5.1 Main Results
Q#1: is the RED policy strong? We measure the win rates of different policies under two test-time command strategies, i.e., a pure NA strategy, which always gives an NA, and an Oracle strategy, which gives "optimal" commands. For the NA strategy, the GRL problem degenerates to the conventional RL setting, so a strong GRL policy should at least achieve comparable performances to the RL policy and substantially outperforms the BC (i.e., Switch) policy. For the Oracle strategy, was scripted to give carefully tuned commands based on the ground-truth game state to instruct the policy to build dominating units according the rock-paper-sccisoors dynamics. Hence, the GRL problem is converted to a fully cooperative game, and following Oracle commands becomes the optimal executor strategy. A strong GRL policy should achieve the highest possible win rate. Meanwhile, the policy with the highest win rate must produce a good command-followie the win rates of different methods with random commands and a "human proxy" commander, which is learned from human data. Since Random and Human Proxy are sub- optimal,, good GRL policy should have substantially lower win rate than the NA case but still outperforms BC (Switch) policy.
behavior. We also report
The results are summarized in Tab. The win rate by RED is the highest under oracle commands and is comparable RL. and Switch without commands (NA). Both RED and Joint policies substantially outperform Switch (BC) policy with sub-optimal commands (Random and Human Proxy), suggesting the conditioned behaviors are improved. Neither RL nor IRL is able make the policy obedient- even under random commands, their win rates remain comparable to the NA case. We also note that commands from human proxy generally yield an even lower win rate than random commands. We remark that learning good human commander model is extremely hard not only due to the limited human command data in the dataset but also because most of the commands are highly sub-optimal.
We empirically notice that lower validation NLL does necessarily yield higher prediction accuracy.
7

---

Command Ignorrat
Hommand-Foolooinng
Test Commands
RL
IRL
Switch
Joint
RED
Oracle (%)
89.3 + 0.7
57.3 3.9
78.7 + 0.9
90.2 + 0.7
92.6 + 0.6 0.6
NA (%)
57.5 + 1.1
45.6 + 2.9
57.5 + 1.1
51.8 = 0.8
57.8 + 1.3
Random (%)
53.3 + 0.4
47.9
12.3 + 0.2
32.3 + 0.3
29.8 0.8
Human Proxy (%)
43.6 + 0.6
48.9
7.8 0.1
11.4 0.7
11.0 = 0.4
Adversarial Oracle (%)
15.9 + 0.7 35.0 2.4
0.2 de 0.1
0.4 0.2
0.9 0.4
Table 1:
Win rates of different policies under various test-time commands. For Oracle and NA, better policy should have a higher reward. For sub-optimal commands, i.e., Random and Human Proxy, an obedient policy should have a much lower win rate than the case of commands (i.e., NA). For Adversarial Oracle, the win rates of command-following policies drop to almost O.
Q#2: is the RED policy grounded? Note that Tab. ] has already provided evidences on the Command-following performance of the RED policy, since the highest win rates can be only achieved by following the Oracle commands. We also implement an "Adversarial Oracle" commander, which always chooses the worst dominating units to build. We can observe that the win rates of Switch, Joint and RED drop to almost O, which suggests that they follow commands even when the commands are divergent from the winning strategies.
Here we present an additional criterion: intuitively, as a command-following policy, its win rate should decrease if "worse" commands are fed; otherwise, if the win rate does not decay, the policy must not follow the commands well. Hence, we interpolate between the NA strategy and the full Random strategy and evaluate the win rate of different policies with different ratios of random commands. The results are shown in Fig. 2 We can observe that the RL policy and IRL policy are clearly not following the commands. Among the remaining policies, RED policy generally produces higher win rate than Switch and Joint, which provides evi- dences that our RED algorithm leads to improved strategies under the fommand-followingg requirement.
0.5
Rate
0.4
RED
Switch
Joint
0.1
Percentage Random Commands Random
Figure 2:
Win rates with an increas- ing amount of random commands. An obedient policy should have de- creasing win rate.
We also try evaluate the command-following ability directly. At the beginning of a game, provide a sequence of commands to guide the policies to build a specific army unit, and evaluate the success rates in 30 steps. The results are listed in Tab. 2 We can observe that the Switch, Joint, and RED policies achieve significantly higher success rates than IRL. This suggests that Switch, Joint, and RED faithfully follow the commands, while IRL tends to be -innoand-ggnorat..
Hommand-Iggnoran
Command-Following
Unit Type
RL
IRL
Switch
Joint
RED
Spearman (%)
98.9 + 0.2
82.6 + 2.7
90.6 00660..9 0.9
97.5 + 0.4
96.0 + 1.4
Swordman (%)
9..2
70.4 5.2
83.9
94.0 1.0 9..2
Cavalry (%)
8..4 0.6
9.4
98.8 0.4 9.2
Dragon (%)
8..
0.2
882 0.8 0.8
Archer (%)
8
2.3
96.3
0.1
9.8
94.6 + 1.5
Catapult (%)
85.3 0.8
2.2 0.9
95.3 + 0.8
0.2
94.6 + 1.9
Table 2: The success rates of building army units given the corresponding commands. is also worth noting that RL still achieves reasonable success rates. This is probably caused by the fact that RL policy is initialized with the BC policy and keeps weak command- followwig ability. Building a single unit at the beginning of a game is relatively easy. However, following commands in complex situation later in the game may be more difficult.
5.2 Out-of-Distribution Commands
As mentioned in Sec. 3.2 we adopt an L.STM encoder to encode arbitrary natural language commands into fixed-length sentence embeddings. During the training process, there are total of 38,558
8

---

different commands in the training set. We also evaluate how the policies perform on Out-of- Distribution (OOD) commands. As shown in Tab, [] we try different ways of adding noise to Oracle commands and report the win rates. We can observe that the agents under noisy oracle commands perform better than Random commands but worse than Oracle. Intuitively, we can observe that the win rate roughly follows "Randon" < "Drop" ~ "Replace" < "Shuufle" ~ "Inserf" < "Oracle". This suggests that the LSTM encoder is sensitive to input words and word order. We also find that stroongfoloowin policies are affected more by noisy commands than weak-following policies, indicating that stronggfollowing policies are more sensitive to the commands. In addition, we also investigate the influence of Out-of- Vocabulary (OOV) words in Appendix D.5, and visualize the command representations in Appendix D.6.
Hommand-Ignorann
Command-Followeer
Noise Type
RL
IRL
Switch
Joint
RED
Random (%)
53.3 + 0.4
47.9
2.8
12.3 0.2
32.3 + 0.3
29.8 + 0.8
Drop (%)
61.9 + 1.0
47.4
3.0
38.8 + 0.6
60.4 0.8
61.1 + 0.6
Replace (%)
59.1 2.4
47.0
+ 2.2
27.6 0.9
+.1
56.7 + 1.8
Insert (%)
2.5 0.7
78.3 - 0.7
0.
8.11
Shuffle (%)
1.8
2.2
68.6 6 1.4
0.1
77.9 0.5
Oracle (%)
0.7
3.9
78.7 - 0.9
9.7
92.6
0.6
Table 3: We add noise Oracle commands. For Drop, we delete 50% of words in each command. For Replace, we replace 50% of the words with random words. For Insert, we insert random words and make the original command twice longer. For Shuffle, we shuffle the words in each command.
5.3 Ablation Study
Training commands. We test the performance of RED with different training commands in Fig. The default RED yields the best strategy, i.e., the highest win rate with NA and oracle commands. Training with random commands or human proxy makes the policy command-ignorant both two variants achieve similar win rates under NA and random commands.
Oracle
NA
Random
Human Proxy
Win Rate
34-
33
31
Iterations
Iterations
Iterations
Iterations
RED
RED w/ Human Proxy
RED w/ Random Commands
Figure 3: Win rates of RED with various training commands. RED has the strongest winning strategy
with NA commands. Both random command and human proxy result in command-ignorigg policy.
Ratio of ||D and | We test different ratios of on-policy samples, i.c. || to the demonstration size, i.e., [D]|. in Tab. We re- port the test-time win rates with oracle and NA commands as well as the constraint satisfaction condition, including both NLL (i.e., L(0; D) from Eq. (2 and the action prediction accuracy. on the validation set. The results show that the validation NL.. can be indeed controlled by tuning the dataset ratio, i.e., more RI. samples consistently yielding a higher NLI. By con- trast, we also find that lower NLL does NOT necessarily lead to a higher action prediction accuracy. But we empirically find that simply setting a 1:1 ratio leads to a sufficiently good policy, which is the default choice for our RED algorithm.
[ : |P]
2:1
1:1
1:0.5
1:0.25
Test Win Rate with Different Commands
Oracle (%)
87.7
92.6
92.0
89.6
NA (%)
54.6
57.8
56.1
57.0
Constraint Satisfaction on Validation Set
NLL
3.15
3.13
3.00
2.92
Accuracy (%)
68.9
68.5
69.0
69.2
Table 4: Ablation study on the ratio of ||Px|l || and
|D]] 0.5 means sub-sampling a half of data from D. The data ratio leads to a controlled NLL.

---

Learned strategy. We identify the five most common categories of commands and measure the validation action prediction accuracy for both RED and BC policies to examine under what conditions the actions are changed noting that RED policy uses BC policy as a warm-start. Fig. 4 shows the categorized accuracy differences with the dashed line denoting the overall mean difference. We can observe that RED policy disagrees with human actions the most under commands belonging to "attack" and "defend" categories, which both represent a highly complicated space of possible behaviors (e.g., which enemy unit to attack, how defend). We believe these action changes are caused by policy improvement. While for commands with a low uncertainty belonging to "mine" and "stop" categories, RED policy simply follows humans. These findings provide intuitive justification in Sec.
arengg
20-
15
-10-
0.0
50"
bald
stpp
Figure 4: Action predic- tion accuracy difference between RED and BC for various command types.
empirical supports
5.4 Human Evaluation
202
We pick 4 policies, i.e., RED, Switch, Joint and RL, and invite 30 college student volunteers under department permission to complete a two-stage study. In the first stage, shuffle the order of the policies and ask each student to keep playing with the 4 policies using any commands. Once the student feels familiar with the game and policies enough, he/she is asked to rank the policies by the level of how they follow commands. In the second part, each student is asked play 2 games per policy, and we measure the average winning rate of different policies.
Humrnn Vote me)
Wit Rate
60
6-
Bwitch
Rank
Polices
Figure 5: Human votes on command- following level. x: policy rank- ing. y: vote percentile.
Figure 6: Human win rates when cooperating with 4 models. 60 games per model.
The results of ranking votes and win rates shown in Fig. [ and Fig. 6 Regarding the votes, the average rank of RED, Switch, Joint and RL are 2.3, 2.4, 2.5, 2.8, respectively. Fig. | visualizes the vote percentage for different policies. We notice very interesting phenomenon that every policy has almost the same sst-rank votes. Knowing that we shuffle the policy order while the students often spend long time the first policy to get familiar with MiniRTS, hypothesize that the students are often biased towards the first policy they encountered. Nevertheless, RED has substantially more votes for the 2nd-rank, indicating better command-following capacity. RL has the most 4th-rank votes, which is consistent with our expectation. Regarding the win rates (Fig. 6). RED outperforms Switch and Joint with clear margin. The win rate of RL comparable with RED. We believe this is due to the fact that RL policy often ignores human commands and simply acts for winning.
6 Conclusion
We tackle a new problem, grounded reinforcement learning (GRL), which aims to learn an agent that can not only get high rewards but also faithfully follow natural language commands from humans. We proposed tractable objective, and developed an iterative RI. algorithm RED, and evaluated the derived policy real-time strategy game MiniRTS. Both simulation and human results show that RED policy achieves high win rates while exhibits strong command-followe
capabilities.
Limitation and social impact. As an initial study on the GRL problem, the RED algorithm is only evaluated on MiniRTS. This is because MiniRTS is the only public environment that both complex (beyond "reaching goals") and provides natural language data. In addition. although we draw connections existing theories in multi-task learning to justify why RED works, the theoretical principle of the underlying optimization process remains an open problem. Finally, even though it in general debatable whether robot should be fully obedient 18 50 58 we believe that in restricted single-agent domains, like playing video games, strictly enforcing fommand-followee behaviors should not result in worse negative social impact than BC or other RI. methods.
Acknowledgement
Yi Wu is supported by 2030 Innovation Megaprojects of China (Programme on New Generation Artificial Intelligence) Grant No. 2021AAA0150000.
10

---

References
[1] Joshua Achiam, David Held, Aviv Tamar, and Picter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[2] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang- -uein Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as I can and not as say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.0 01691, 2022.
[3] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-languun navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674-3683, 2018.
[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433. 2015.
[5] Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions actions. Transactions of the Association for Computational Linguistics, 1:49-62, 2013.
[6] Monica Babes, Vukosi N. Marivate, Kaushik Subramanian, and Michael L. Littman. Appren- ticeship learning about multiple intentions. In ICML, pages 897-904, 2011.
[7] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. Learning understand goal specifications by modelling reward. In International Conference on Learning Representations, 2019.
[8] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175, 2010.
[9] Shai Ben-David and Reba Schuller Borbely. A notion of task relatedness yielding provable multiple-task learning guarantees. Machine learning. 73(3):273-28) 2008.
[10] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota with large scale deep reinforcement learning. arXiv preprint arXiv: /et.: 2019.
[11] Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May. Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-8735. Association for Computational Linguistics, November 2020.
[12) Micah Carroll, Rohin Shah, Mark Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
[13) Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In AAAI, 2018.
[14] David Chen and Raymond Mooney. Learning to interpret natural language navigation instruc- tions from observations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 859-865- 201
11

---

[15] Maxime haria, Thien Huu Nguyen, and Yoshua Bengio. BabyAI: First steps towards grounded language learning with human in the loop. In International Conference on Learning Representations, 2019.
Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Sa-
[16] Kurtland Chua, Qi Lei, and Jason D Lee. How fine-tuning allows for effective meta-learning. Advances in Neural Information Processing Systems, 34, 2021.
[17] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE international conference on robotics and automation (ICRA), pages 4693-4700. IEEE, 2018.
[18) Derek Cormier, Gem Newman, Masayuki Nakane, James E. Young, and Stephane Durocher. Would you do as a robot commands? an obedience study for human-robot interaction. In International Conference Human-Agent Interaction, 2013.
[19] Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural modular control for embodied question answering. In Conference on Robot Learning, pages 53-62. PMLR, 2018
[20] Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-tast learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computationnal Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 1723-1733 1732, 2015.
[21] Yunshu Du, Wojciech M Czarnecki, Siddhant M Jayakumar, Mehrdad Farajtabar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. The Continual learning Workshop at NewrIPS, 2018.
[22] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg- Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker- follower models for vision-and-language navigation. Advances in Neural Information Process- ing Systems, 31, 2018
[23] Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. In International Conference on Learning Representations, 2019.
[24] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
(25) Yang Gao, Huazhe Harry Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Rein- forcement learning from imperfect demonstrations. International Conference on Learning Representations, Workshop Track, 2018.
[26] Denise Y Geiskkovitch, Derek Cormier, Stela H Seo, and James E Young, Please continue, we need more data: an exploration of obedience robots. Journal of Suman-Robo Interaction. 9(19,82-99, 2016.
[27] Vinicius G Goecks, Gregory M Gremillion, Vernon J Lawhern, John Valasek, and Nicholas R Waytowich. Integrating behavior eloning and reinforcement learning for improved performance in dense and sparse reward environments. arXiv preprint arXiv: 1rXiv:1910.04239) 2019.
[28] Dylan Hadfelld-Mnnel,, Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29, 2016.
[29] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech M. Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wain- wright, Chris Apps, Demis Hassabis, and Phil Blunsom. Grounded language learning a simulated 3d world. ArXiv, hls/1700.06551. 2017.
[30] Karl Moritz Hermann, Mateusz Malinowski, Piotr Mirowski, Andras Banki-Horvath, Keith Anderson, and Raia Hadsell. Learning to follow directions in street view. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11773-11781, 2020.
12

---

[31] Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instructioonfoolloiin with deep reinforcement learning via leammig from text. CoRR, abs/2005.09382, 2020.
[32] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[33] Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. Hierarchical decision making by generating and following natural language instructions. Advances in neural information processing systems, 32, 2019.
[34] Michaela Jänner, Karthik Narasimhan, and Regina Barzilay. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics, 6:49-61, 2018.
[35] Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language an abstraction for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems. 32, 2019.
[36] Siddharth Karamcheti, Megha Srivastava, Percy Liang, and Dorsa Sadigh. LILA: Language- informed latent actions. In 5th Annual Conference on Robot Learning, 2021.
[37] Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. Toward understanding natural language directions. In 2010 5th ACM/IEEE International Conference on Sluman-Robor Interaction (HRI), pages 259 356.266.12 IEEE, 2010.
[38] Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack learning environment, Advances Neural Information Processing Systems, 33:767 1-7684, 2020.
[39] Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. Deal no deal? end- to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443-2453, 2017.
[40] Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. Advances in neural information processing systems, 32, 2019.
[41] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2020.
[42] Yuchen Lu, Soumye Singhal, Florian Strub, Olivier Pietquin, and Aaron Courville. Supervised seeded iterated learning for interactive language learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguistics.
[43] Yuchen Lu, Soumye Singhal, Florian Strub, Olivier Pietquin, and Aaron C. Courville, Counter- ing language drift with seeded iterated learning. In ICML, 2020.
[44] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems XVII, 2021.
[45] Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. Def. 2(6):4, 2006.
[46] Cynthia Matuszek. Grounded language learning: Where robotics and NLP meet (invited talk). In Proceedings of the International Joint Conference on Artificial Intelligence, 2018.
[47] Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. Learning to parse natural language commands to a robot control system. In Experimental robotics, pages 403-415. Springer, 2013.
[48] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning, Journal of Machine Learning Research, 17(81)11-32, 2016.
13

---

[49] Harsh Mehta, Yoav Artzi, Jason Baldridge, Eugene Ie, and Piotr Mirowski. Retouchdown: Adding touchdown to streetlearn as shareable resource for language grounding tasks in street view, 2020.
[50] Smitha Milli, Dylan Madeield-Menell, Anca Dragan, and Stuart Russell. Should robots be obedient? Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 4754-4760, 2017.
[51] Aditya Mogadala, Marimuthu Kalimuthu, and Dietrich Klakow. Trends in integration of vision and language research: A survey of tasks, datasets, and methods. Journal of Artificial Intelligence Research, 71:1 181-111311111 2021.
[52] Khanh Nguyen, Dehadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12527-12537, 2019.
[53] Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In Interna- tional Conference on Machine Learning, pages 3878-3887. PMLR, 2018.
[54] Lerrel Pinto and Abhinav Gupta. Learning to push by grasping: Using multiple tasks for effective learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 2161-2166 IEEE, 2017.
[55] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration deep rein- forcement learning, arXiv preprint arXiv:1910.01 7:1, 2019.
[56] Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. A benchmark for systematic generalization in grounded language understanding. Advances in neural information processing systems, 33:19861- 2020.
[57] Stuart Russell. Human compatible: Artificial intelligence and the problem of control. Penguin, 2019.
[58] Tanja Schneeberger, Sofie Ehrhardt, Manuel S Anglet, and Patrick Gebhard. Would you follow my instructions if I was not human? Examining obedience towards virtual agents. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 1-7. IEEE, 2019.
[59] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[60] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. CLIPort: What and where pathways for robotic manipulation. Conference on Robot Learning, pages 894-906. PMLR, 2022.
[61] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740-10749, 2020.
[62] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021.
[63] Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning, pages 91 9120-9132. PMLR, 2020.
[64] Lue Steels and Manfred Hild. Language grounding robots. Springer Science Business Media, 2012.
[65] Simon Stepputtis, Joseph Campbell, Mariano Phielipp. Stefan Lee, Chitta Baral, and Heni Ben Amor Language-conditinn conditiondd imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 13:13139-13150 2020.
14

---

[66] Peter Stone, Gal A Kaminka, Sarit Kraus, and Jeffrey S Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. Twenty-Fourde AAAI Conference on Artificial Intelligence, 2010.
[67] Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6629-6638, 2019.
[68] Nicholas Waytowich, Sean L Barton, Vernon Lawhern, Ethan Stump, and Garrett Warnell. Grounding natural language commands to StarCraft II game states for narration-guided re- inforcement learning. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, volume 11006, page 110060S. International Society for Optics and Photonics, 2019.
[69] Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1-191, 1972.
[70] Mark Woodward, Chelsea Finn, and Karol Hausman. Learning to interactively learn and assist. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 2535-2543, 2020.
[71] Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3D environment. International Conference on Learning Representations, Workshop Track, 2018.
[72] Tsung-Yes Yen Yang. Justinian Rosca, Karthik Narasimhan, and Peter Ramadge. Projection- constrained policy optimization. In International Conference Learning Representations, 2020.
[73] Tsung- Yen Yang. Justinian Rosca, Karthik Narasimhan, and Peter Ramadge. Accelerating safe reinforcement learning with constraint-mismatchn baseline policies. International Conference on Machine Learning, pages 11795-18077 PMLR, 2021.
[74] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[75] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, Advances in Neural Information Processing Systems, 33:5824-5836, 2020.
[76] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- Meta-World: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1100 100. PMLR, 2020.
[77] Luke Zettlemoyer and Michael Collins. Online learning of relaxed CCG grammars for parsing to logical form. In Proceedings of the 2007 Joint Conference on Empirical Methods Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLA) pages 678-687, Prague, Czech Republic, June 2007. Association for Computational Linguistics.
[78] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3713-3722, 2019.
[79] Li Zhou and Kevin Small. Inverse reinforcement learning with natural language goals. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11116-11124, 2021.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section.
(b) Did you describe the limitations of your work? [Yes] See Section. 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] In supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix.B B.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix.B.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [Yes] See Appendix.A.
(c) Did you include any new assets either in the supplemental material or as a U
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [Yes] See Appendix.A.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] See Appendix.A.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] See Appendix.F.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [Yes] See Appendix.E.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] See Appendix.E.
16


Reviews:
[{"review_id": "oy2OY0UxRDr", "reviewer": "Reviewer_8YRG", "summary": "This paper studies the problem of building reinforcement learning agents constrained by human demonstrations in the real-time strategy game MiniRTS.", "questions": "1. In the human study, how did the author evaluate the level of skills of participants and how they determined when the participants were \"familiar\" (line 353) with the game?  \n2. Could the authors clarify the difference between the framework GRL they proposed and existing language grounded approached? The manuscript would benefit from the authors addressing the difference of their setting with existing language grounded approaches. I do not believe that the complexity of MiniRTS as an environment is a sufficient argument to dismiss previous work.", "limitations": "The author address some limitations of their work and the setting discussed in the paper is interesting from its potential societal impact of having RL agent that trade-off following human commands with the optimality of the actions it executes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths** The paper proposes an interesting approach, REinforced demonstration Distillation (RED), to find RL agents that trade off following possibly suboptimal human commands with finding an optimal solution to the MiniRTS environment. I think this is an interesting setting, and the manuscript presents the problem and related work in a clear and concise manner. The baselines the authors selected are appropriate for the problem set up\n\n**Weaknesses**\nWhile the manuscript claims that MiniRTS is harder than following natural language instructions in some grid world navigation environments (e.g., [1]). For starters, the problem of training from natural language seems harder for two reasons. First, understanding natural language instructions seems a harder problem than the command setup proposed in the paper. Furthermore, the problem that this previous navigation work analyzes is usually under sparse rewards (success/fail) while in the setting proposed in the paper there are dense demonstrations that may alleviate the complexity of the task.\n\nIn terms of novelty, constraining RL with demonstrations as has been extensively studied in the literature [2,3] and the architectural challenges for solving the language-grounded task were introduced in MiniRTS [4].\n\n[1] Chevalier-Boisvert et al. BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning. 2018\n[2] Yang et al. Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies. 2020\n[3] Goecks et al. Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments\n[4] Hu et al. Hierarchical Decision Making by Generating and Following Natural Language Instructions\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bEVeKQvUl12", "reviewer": "Reviewer_9kih", "summary": "This paper studies Grounded RL, where an agent is trained to obey natural language commands while still trying to attain the highest reward. The paper proposed to iterate between a pure RL loss and a behavior cloning loss, where the BC loss takes trajectories from both human demo and from the winning ones in the current on-policy sample queue.\n\nThe method is evaluated on the MiniRTS env and dataset and it is shown that the method can balance between high winning rate when no command is given, and low winning rate when the command is random. \n", "questions": "* Did you run any experiments to see whether the agent is able to follow certain commands with clear goals? E.g. “build two archers” is a clearly defined one and easy to evaluate if supported by the MiniRTS api. \n* How did the “switch” policy do in terms of obeying the commands, especially the not-NA commands? It would be good to compare it with that of RED, to show that both can obey commands, but RED wins more – which is a central goal you are trying to achieve, iiuc.\n\nComments: \n* It will be interesting to see an “adversarial oracle” where they always choose the worst dominating units to build. \n* It is interesting that “human proxy” almost always performs worse than random. ", "limitations": "The ratio between ||D_k|| and ||D|| and the threshold delta are dataset/env specific hparams which require extra evaluation to tune, and which will affect the command-obeying ability. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* I appreciate the discussion around why the Lagrange multiplier is not suitable.\n* The “random commands” is a good sanity check to see if the agent is affected by bad commands. \n* The assumptions and the alternatives are clearly stated (e.g. l151-152, l243-244, etc.)\n\nWeaknesses:\n* Although often observed in practice, as mentioned in l204-206, the fact that the policy can generalize at all beyond BC for any not-NA command relies on the architecture of the policy network. If that is the core, then this paper did not study the condition for which such generalization occurs. And it’s such a core assumption of the proposed method, that to leave it understudied makes me a bit uneasy.\n* While I appreciate the social impact section, “should not result in any negative social impact” is a bit too extreme. One can easily imagine military uses where humans provide data in a simulator and the policy is later deployed in the real world. A stronger command-following RL agent will have a negative impact there, allowing better performance compared to an RL agent trained from scratch (which doesn’t work in the MiniRTS case). You may be able to argue that it has no worse social impact than say BC or other RL methods, but saying “any” is too strong.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_LGP9eHt4ae", "reviewer": "Reviewer_WZeZ", "summary": "Paper introduces a new task called “Grounded Reinforcement Learning” in which agents must be reward-maximizing with respect to some high-level language “constraint” (given by another agent/human). The authors demonstrate that existing vanilla RL, constrained RL (CRL), and behavior cloning (BC) fail to perform well both with and without commands in the MiniRTS environment (an example of this problem setup).\n\nTo solve this problem setting, the authors propose a new method RED - Reinforced Demonstration Distillation, which repeatedly alternates between pure RL and BC updates.  The authors demonstrate that RED is both the strongest and most grounded policy. The authors provide, through ablations, compelling arguments. ", "questions": "1) It is unclear how this approach is different from having standard RL with a combined state space over both command and game state.  As the command representation is not learnt - it seems unclear how this actually is different to a standard RL problem.\n\n2) Can more justification be given to how difficult the opponent you play with is? Can you play with a harder agent to more clearly demonstrate the advantages of your method?\n\n3) It is really unclear how the command token is meaningfully different to having an additional dimension within the game state. Can some justification be given to how either this representation is learnt or has any assemblance of language (is it compositional, have semantic or pragmatic properties).\n", "limitations": "Addressing the above questions would be useful. \n\nI also believe actually using a language model would take great steps in demonstrating the effectiveness of your approach. \n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nIntroduces (somewhat) of a new problem statement\n\nClearly written \n\nHuman Evaluation is compelling\n\nImpactful work from a safety perspective. \n\nWeaknesses:\n\nMy largest criticism is the use of the word \"language\" and i think this undermines the entire approach - the current representation treats commands as one-hot representations. These are not learnt, shaped or any different to the state space. This has no standard components which i would consider natural language, such as pragmatics, syntax and semantics.\n\nRED algorithm does not seem novel (effectively mixture of two)\n\nResults are not significantly better than baselines.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QX0cW8KDS2H", "reviewer": "Reviewer_wnXo", "summary": "This paper takes on the challenge of both playing a game as optimally as possible, while following instructions and learning from demonstrations (both of which may be sub-optimal). The paper formulates this problem as a joint optimization, and tackles it by alternating steps of RL training with steps of distilling both demonstrations and successful RL trajectories. The paper evaluates this approach on the MiniRTS game, and evaluates against a range of baselines on various aspects such as rate of winning under different command types, how humans rank the policy’s instruction following, etc.", "questions": "See above, but briefly:\n* Could the authors demonstrate more directly how well the agents follow commands—particularly in cases where they commands are strongly divergent from the winning policy?\n* What do the authors think might be the risks of the proposed algorithm?", "limitations": "On the whole, I think this paper could be a useful contribution. The most important changes from my perspective would be (see above for full details):\n* The claim that the learned policy is grounded is not particularly well supported. Experiments on how well the agents are actually following commands would be most important for improving my rating.\n* There are risks to trying to minimally follow commands while pursuing another objective. Discussing these shouldn't be a major change.\n\nOther limitations that I don't think are as critical to address, but would be helpful:\n* Only evaluating on a single game makes it difficult to know how general the advantage is.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Post-response update\n-----------\n\nThe authors have addressed some of my main concerns, and more thoroughly evaluated their method. I believe these new experiments could be better integrated in to the main text, and the paper is still limited by experimenting only with a single environment, but I am increasing my score in accordance with the improvements.\n\n\n\nOriginal review\n--------------\n\n\nStrengths:\n* An interesting and well-motivated problem space and proposed methods.\n* In-depth experiments evaluating different perspectives on the issues raised in one domain.\n* Good ablations and baseline conditions.\n* Generally well written and clear. \n\nWeaknesses:\n*The claim that the learned policy is grounded is not directly supported, only through circumstantial evidence that (while helpful) is not as direct as it could be.\n  - The fact that win rates decrease with random commands is very weak evidence that the commands are being followed. The fact that humans rank the RED policy slightly higher than the rest is a little better, but the effect is weak (as noted in the paper there is an even distribution over the first rank). The statistical significance of the effect is not quantified, but there would almost certainly not be a significant difference between RED and the Switch or Joint baselines. \n  - To address this concern, it would be ideal to actually evaluate how well the agent follows commands. For example, set up a situation like Fig. 1E, give the agent a command like “Retreat”, and then evaluate whether the agent actually moves its units away or simply continues attacking, and how long it maintains this behavior. This would be far stronger, direct evidence that the agent is actually following commands (rather than e.g. doing nothing, issuing resource gathering commands without actually retreating, etc.).\n  - Performing these tests in *precisely* situations where the RL policy would do something very different would be the strongest test of behavior.\n* Only evaluating on a single game makes it difficult to know whether the conclusions are generalizable; it would be ideal to perform experiments in 1-2 other settings as well.\n* The general approach to the prior literature could be improved. For example:\n  - The paper argues that “BC and IRL algorithms assume that the demonstrations are optimal”—however, a variety of prior works have addressed this problem in various ways, both from the perspective of IRL (e.g. https://proceedings.mlr.press/v97/brown19a.html) and from BC-like approaches conditioned on rewards (https://arxiv.org/abs/1912.02875), or just by using BC to initialize RL training (e.g., https://www.nature.com/articles/nature16961). While not all these perspectives would necessarily address the issue of also following commands, they are not incompatible with it, and the paper should discuss how such approaches might relate to (or complement) the proposed technique.\n  - Relatedly, prior work on agents that follow human instructions (particularly https://arxiv.org/abs/2012.05672) has relied on a combination of BC, RL, and IRL to achieve better performance than BC alone; this would be useful to discuss.\n* It seems to me that there is a fundamental tension between the goals of strictly following human commands, and winning the game. The proposed approach appears to pursue a policy that effectively tries to distort all human commands towards its own objective (i.e., to follow the command in the way that is most likely to lead to winning, as in Fig. 1C vs. 1D). Phrased in this way, it seems that there are clear alignment concerns about this approach. These should be discussed as a tradeoff/risk.\n  - These concerns relate to the above questions about how well the agent is following commands—if the goal is for the agent to actually strictly follow human commands, it should not e.g. follow a retreat order for one second and then go right back to attacking.\n  - It seems to me that it would be *far* safer to have a system that follows human commands first and foremost—i.e., a system that is “interruptable” (https://deepmindsafetyresearch.medium.com/building-safe-artificial-intelligence-52f5f75058f1 and https://www.ijcai.org/Proceedings/2017/0032.pdf)\n  - This concern is *somewhat* ameliorated by the fact that the win rates under sub-optimal commands are much lower, but it depends on how frequently the commands are issued in these conditions.\n  - Regardless, these issues should be discussed.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DtnQw2ZXDJD", "reviewer": "Reviewer_oT9Z", "summary": "This work focuses on grounded reinforcement learning. Generally, it means finding a conditional RL policy that can perfectly response to high-level command. \n\nIn this paper, the authors proposed a constraned RL algorithm RED to (1) train an agent that is capable of winning strategy games and (2) follow most human instructions. Concretely, RED uses RL to learn explorative policy while uses BC to imitate human demonstration.\n\nExperiment results suggest RED achieves command-following policies and higher winning rate.", "questions": "### Questions on method\n\n\nSec 4.3 says this work uses Lagrangian multiplier to solve the constrained optimization problem in Eq2. It turns out that, in my understanding, this is simply a weighted loss. A formal Lagrangian method should contain update process of the $\\beta$, which is just ignored by this paper (or I miss, please point out). The later \"iterative solution\" also breaks the so called \"Lagrangian method\". \n\n---\n\nLine 240 and Appendix A.2: Could you discuss more details on how you deal with action space? What does the \"0/1 action for each unit\" means? Who does the selection?\n\n---\n\nHow to insert a NA command? IIUC, there exist a fixed set of possible commands right? So NA command should be just one more command in that set. You create embedding for each command in the set and than pass the embedding to the neural network.\n\n---\n\n\n\n\n\n### Questions on experiments\n\nI can't understand table 1. IIUC, these four baselines have the policies that do not take command as input. Why they have different performance in Table 1 when applying different test commands? Please describe how you turn them as conditional policies. \n\nFor example, since RL is trained without any objective to conduct distinguishable behaviors under different commands, why inserting different test commands will lead to different behavior?\n\n---\n\nI don't think Table 1 is convincing, especially when taking STD into account. \n\nLine 288: \"Both RED and Joint policies substantially outperform Switch (BC) policy with sub-optimal commands (Random and Human Proxy), suggesting the conditioned behaviors are improved.\" This is a overclaimed statement. I can't draw this conclusion from Table 1.\n\n---\n\nI like Fig 2, which shows the dependency between commands and behaviors. Many \"conditional RL\" methods fail to prove that the conditional policy has casuality to the input command. This figure also shows that RED has much better \"casuality\" than RL.\n\nMy problem is that, in Figure 2, RL in 0% random commands achieves ~57% win rate. Does this value align with the RL performance with NA test command in Table 1?\n\nAgain, if RL is trained compelety without human command, then (1) where you can insert the command to the policy? and (2) why RL performance changes with the percentage of random commands?\n\n---\n\n\nFigure 6 says RL policy achieves similar win rate compared to RED. IIUC, RL policies are not conditioned on any human command. This result suggest that a human subject cooperates with RED trained conditional policy will lead to RL-comparable performance? This seems to be a violation to Table 1 \"Oracle\" result, which might lead to \n\n---\n\nThere is only one real gameplay footage provided (less than 2 minutes), with only 6 commands given, which is far less than convincing. Meanwhile, the commands seemed to be programmed/copy-pasted (I did not see any input lag), is there any real human intervention involved in recoding this video?\n\n---\n\nThis game seems to be very complicated. Does your first intention of the application of this algorithm to be this high-level game? I understand the miniRTS environment is the only one that fitted the algorithm's need, but if you have alternatives (even if the performance is weak), would you also like to present them in the paper? Having diversity is a huge plus to your paper.\n\n\n\n\n\n\n\n\n### Questions on paper\n\n\n\n\n### Other questions or concerns\n\n\n\nThe [google drive link](https://drive.google.com/file/d/129ZV9CU6zUnp5B4v0p91Xaae8-M5zxHy/view) provided is **not anonymous**. This is a violation of the NeurIPS Code of Conduct (double-blind reviewing policy). Please action ASAP.\n\n---\n\nThis paper has experiments involving 30 college students experiments, are attendees aware of their rights and responsibilities? A more explicit Ethics Statement should be included.\n\n---\n\nConditional BC is not a new idea. Please refer to this work: End-to-end Driving via Conditional Imitation Learning\n\n---\n\nAppendix D, as said in the checklist, has no information on human experiment. You moved them to Appendix E.\n\n---\n\nThe website given in the appendix is very informative and I like the real-time gameplay footage, but there are several problems I want to point out:\n- Code link is broken.\n- Many grammar errors, will not be present here (plural or singular, tense, etc) \n", "limitations": "The authors provide limitation section.\n\nGRL is an interesting field. How to improve human-AI cooperation in test-time, how to learn a better human proxy, and how to further encourage, illustrate, quantify the causity between command and behaviors might be future directions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n### Strengths\n\n* The idea is concise. Use BC to learn human behavior while use RL to explore better solution.\n* Visualization and videos provided are mostly clear and contain necessary information.\n* Fig1 is illustrative and helpful to explain what is grounded reinforcement learning. (It would be better if Fig 1 has no background color and in PDF format)\n* Code is released.\n\n### Weaknesses\n\n* This work assumes reward from interactive environment. (limitation)\n* The code has no detailed guidance and documentation.\n* Though I like Figure 2 and other experiments, the main experiment results in Table 1 is confusing and non-convincing.\n* The idea is not novel. Conditional imitation learning is a long lasting topics. Refer to \"End-to-end Driving via Conditional Imitation Learning\"\n* Many details are missing or confusing, such as Table 1 and Lagrangian method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept