Improving Policy Learning via Language Dynamics Distillation
Authors: Victor Zhong
Jesse Mu
Luke Zettlemoyer
Edward Grefenstette
Tim Rocktäschel
Venue: NeurIPS 2022 Accept (2022)
Keywords: language grounding
reinforcement learning
reading to generalize

Abstract: Recent work has shown that augmenting environments with language descriptions improves policy learning. However, for environments with complex language abstractions, learning how to ground language to observations is difficult due to sparse, delayed rewards. We propose Language Dynamics Distillation (LDD), which pretrains a model to predict environment dynamics given demonstrations with language descriptions, and then fine-tunes these language-aware pretrained representations via reinforcement learning (RL). In this way, the model is trained to both maximize expected reward and retain knowledge about how language relates to environment dynamics. On SILG, a benchmark of five tasks with language descriptions that evaluate distinct generalization challenges on unseen environments (NetHack, ALFWorld, RTFM, Messenger, and Touchdown), LDD outperforms tabula-rasa RL, VAE pretraining, and methods that learn from unlabeled demonstrations in inverse RL and reward shaping with pretrained experts. In our analyses, we show that language descriptions in demonstrations improve sample-efficiency and generalization across environments, and that dynamics modeling with expert demonstrations is more effective than with non-experts.

Improving Policy Learning via Language Dynamics Distillation
Victor Zhong!? Jesse Mu?, Luke Zettlemoyer!?, Edward Grefenstette and Tim Rocktäschel*
University of Washington
Meta AI Research
3 Stanford University
University College London Cohere
Abstract
Recent work has shown that augmenting environments with language descriptions improves policy learning. However, for environments with complex language abstractions, learning how to ground language to observations is difficult due sparse, delayed rewards. We propose Language Dynamics Distillation (LDD), which pretrains a model to predict environment dynamics given demonstrations with language descriptions, and then fine-tunes these language-aware pretrained representations via reinforcement learning (RL). In this way, the model is trained to both maximize expected reward and retain knowledge about how language relates to environment dynamics. On SII.G, a benchmark of five tasks with language de- scriptions that evaluate distinct generalization challenges on unseen environments (NetHack, ALFWorld, RTFM, Messenger, and Touchdown). LDD outperforms tabula-rasa RL, VAE pretraining, and methods that learn from unlabeled demon- strations in inverse RL and reward shaping with pretrained experts. In our analyses, we show that language descriptions in demonstrations improve sumple-efficieng and generalization across environments, and that dynamics modeling with expert demonstrations more effective than with non-experts.
1 Introduction
Language is powerful medium that humans reason about abstractions itss compositionality allows efficient descriptions that generalize across environments and tasks. Consider an agent that follows instructions to clean the house (c.g. find the dirty dishes and wash them). In tabula-rasa reinforcement learning (RL), the agent observes raw perceptual features the environment, then grounds these visual features to language cues to learn how to behave through trial and error. In contrast, we can provide the agent with language descriptions that describe abstractions which are present in the environment (e.g. there is sink to your left and dishes a table to your right), thereby simplifying the grounding challenge. Language descriptions of observations occur naturally in many environments such text prompts in graphical user interfaces [Liu al. 2013 dialogue [He et al.) 2018), and interactive games [Küttler et al. 2020). Recent work has also shown improvements in visual manipulation (Shridhar et al. [021] and navigation [Zhong al. 2021 Tam et al.) [022] by captioning the observations with language descriptions. Despite these gains, learning how to interpret language descriptions is difficult through RL, especially environments with complex language
abstractions Zhong et al. 2021
Corresponding author Victor Zhong vzhong@cs. washington edu
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Environment language observations
Observations from expert demonstrations without actions
Record observations
and rich Tule
Next-obs prediction
Dynamics modeling
nesime sesent.
Teacher dynamics model
Policy learner
Initialize
fine-tuning
Final
policy
Distill
Policy learning
Figure I: Language Dynamics Distillation (LDD). LDD uses cheap unlabeled demonstrations to learn dynamics model of the environment, which is used to initialize and distill grounded representations into the policy learner. During the dynamics modeling phase (purple), train a teacher model to predict the next observation given prior observations using unlabeled demonstrations. the policy learning phase (red), we initialize model with the teacher and distill intermediate representations from the teacher during reinforcement learning. The traditional policy learning loop is shown in green. LDD- speciiic components shown in blue.
We present Language Dynamics Distillation (LDD), a method that improves RL by learning a dy- namies model cheaply obtained unlabeled (i.e. no action labels) demonstrations with language descriptions. When learning how to use language descriptions effectively, one central challenge is how disentangle language understanding from policy performance from sparse, delayed re- wards. Our motivation is to learn initial language grounding via dynamics modeling from an offline dataset, away from the credit assignment and non-stationarity challenges posed by RL. While labeled demonstrations that tell the agent how to to act each situation are expensive to collect, for many environments one can cheaply obtain unlabeled demonstrations (e.g. videos of experts performing the task) Yang et al., 2019 Stadic et al., 2017). Intuitively, LDD exploits these unlabeled demonstrations learn how to associate language descriptions with abstractions in the environment. This knowledge is then used to bootstrap and more quickly learn policies that generalize to instructions and manuals new environments. Given unlabeled demonstrations with language descriptions (e.g. captions of scene content), we first pretrain the model to predict the next observation given prior observations, similar to language modeling. copy of this model stored as a fixed teacher that grounds language descriptions to predict environment dynamics. We then train model with RL, while distilling inter- mediate representations from the teacher to avoid catastrophic forgetting of how interpret language descriptions for dynamics modeling. In this way, the model learns to both maximize expected reward while retaining knowledge about how language descriptions relate to environment dynamics.
We evaluate LDD on the recent SILG benchmark Zhong et al. [021]. which consists of five diverse en- vironments with language descriptions including NetHack Kütter al. 2020 ALFWorld (hridhhr
et al. 2021 RTFM ZZhong et al. 2020 Messenger (Hanjie et al. 2021 and Touchdown CChnn et al.
These environments present unique challenges in language grounded policy-learinng across complexity of instructions, visual observations, action space, reasoning procedure, and generalization. By learning dynamics model from cheaply obtained unlabeled demonstrations, LDD consistently outperforms reinforcement learning with language descriptions both in terms of sample efficiency and generalization performance. Moreover, we compare LDD to other techniques that inject prior knowl-
2018
language-grounded
edge in VAE pretraining [Kingma and Welling 2013 inverse reinforcement learning HHnna and

---

Stone 2017
Torabi et al. 2018 Guo et al, 2019 and reward shaping with pretrained expert [eerll al. 2017 LDD achieves top performance on all environments terms of task completion and reward. In addition to comparing LDD to other methods, we ablate LDD to quantify the effect of language observations in dynamics modeling, and the importance of dynamics modeling with expert demonstrations. On two environments where we can control for the presence of language descriptions (NetHack game messages and Touchdown panorama captions), we show that language descriptions improve sample-efficiency and generalization. Finally, across all environments, we find that dynamics modeling with expert demonstrations is more effective than with non-expert rollouts.
2
Related Work
Learning by observing language. Recent work studies generalization to language instructions and manuals that specify new tasks and environments. These settings range from photorealistic/3D
navigation [Anderson et al. 2018 Chen et al. 2018 Ku et al.
2020 Shridhar et al. (020 to multi-
hop reference games Narasimhan al. 2015 Zhong al. 2020 Hanjie et al. 2021 We use
collection of these tasks to evaluate LDD. There is also work where understanding language is not
necessary to achieve the task, however its inclusion (e.g. via captions, scene descriptions) makes learning more efficient. Shridhar et al. [202T] show that one can quickly learn policies in a simulated kitchen environment described in text, then transfer this policy the 3D visual environment, Zhong similarly transform photorealistic navigation to symbolic form via image segmentation, then learn policy that transfers to the original photorealistic setting. In work concurrent to ours, Tam et al. [2022 generate oracle captions of observations for simulated robotic control and city navigation, which improve policy learning. LDD is complementary to these-in addition incorporating language descriptions as features, we show that learning dynamics model from unlabeled demonstrations with language descriptions improves sample efficiency and results better policies.
et al. 2021 []
Imitation learning from observations. There is prior work on model-free as well as model-based imitation learning from observations. Model-free methods encourage the imitator to produce state distributions similar to those produced by the demonstrator, for example via generative adversarial learning Merell et al. (017] and reward shaping (Kimura et al., 2018). In contrast, LDD only requires intermediate representations extracted from an expert dynamics model on states encountered by the learner, which are cheaper to compute than rollouts from an expert policy. Model- approaches learn dynamics models that predict state-transitions given the current state and an action. [Hanna and Stone 2017] learn an inverse model to map state-transitions to actions, which is then used to annotate unlabeled trajectories for imitation learning. Edwards et al. 2019] learn a forward dynamics model that predicts future states given state and latent action pairs. In contrast, LDD does not assume priors over the action space distribution. For instance, on ALFWorld, our method works even though is impossible enumerate the action space. In our experiments, we extend model-fres reward shaping and model-based inverse dynamics modeling to account for language descriptions and compare LDD to these methods.
Representation learning in RL. In representation learning for RL, the agent learns representations of the environment using rewards and objectives based on the difference between the state and prior states (Strehl and Littman| 2008), raw visual observations (aderberg et al. 2017). learned agent representations [Raileanu and Rocktäschel] 2020], and random network observations [Burda et al.) 2019). In intrinsic exploration methods [Raileanu and Rocktäschel 2020 Burda et al. 2019), , the training objective encourages dissimilarity (e.g. in observation/ sttaoe space) to prior agent experi- ence so that the agent discovers novel states. Unlike intrinsic exploration, the distillation objective in Language Dynamics Distillation encourages similarity to expert behaviour, as opposed to dissim- ilarity prior agent experience. In reconstruction based representation learning methods (Strchl and Littman 2008 Jaderberg et al. 2017]. the training objective encourages the agent to learn intermediate representations that also capture the dynamics and structure of the environment by reconstructing the observations (e.g. predicting what objects are in scene). Language Dynamics Distillation similar to reconstruction methods for representation learning, however unlike the latter, the dynamics model in LDD is trained on trajectories obtained from an expert policy as opposed to the agent policy. Language Dynamics Distillation complementary to intrinsic exploration methods and to reconstruction based representation learning methods.
3

---

3 Language Dynamics Distillation
Recent work improves policy learning by augmenting environment observations with language de- scriptions [Shridhar et al., 2021 Zhong et al. 2021 Tam et al. 2022 For environments with complex language abstractions, however, learning how associate language to environment observations is difficult through RL due to sparse, delayed rewards. In Language Dynamics Distillation (LDD), we pretrain the model on unlabeled demonstrations (i.e. no annotated actions) with language descrip- tions to predict the dynamics of the environment, then fine-tune the language-aware model via RL. LDD consists of two phases. In the first dynamics modeling phase, pretrain the model predict future observations given unannotated demonstrations. We store copy of the model as a fixed teacher that has learned grounded representations useful for predicting how the environment behaves under an expert policy. In the second reinforcement learning phase, we fine-tune the model through policy learning. while distilling representations from the teacher. This way, the model is trained to both maximize expected reward and retain knowledge about the dynamics of the environment. Fig iillustattes the components of LDD.
3.1 Background
Markov decision process. Consider a MDP M (S, (.r,,. Here, S and A respectively are the discrete state (e.g. language goals, descriptions, visual observations) and action spaces of the problem. P(s (st, 0e) is the transition probability of transitioning into state se+1 by taking action a, from state 8t. r(s, a) is the reward function given some state and action pair. 7y is discount factor to prioritize short-term rewards.
Actor-critic methods for policy learning, In RL, we learn a policy <(s; 0) that maps from observations to actions æ : - A. Let R(T) denote the total discounted reward over the trajectory The objective is to maximize the expected reward J- (0) = E,, [R(T]] following the policy by optimizing its parameters 0. For trajectory length T, the policy gradient
[R(+)
T
G.V C..VloogTo ((a,, (.v bi) [[_
T
VE, ((()] = Ex R(T))
Σ
V log (ac, si) = E
Vlogm(ar,
(I)
where G. = -x Fx4.4+1 is the return or discounted future reward at time I. We consider the actor-critic family of policy gradient methods, where critic is learned to reduce variance in the gradient estimate. Let (x) = E- = s] denote the state value, which corresponds to the expected returns by following the policy from state 8. Actor critic methods estimate the state value function by learning another parametrized function V to bootstrap the estimation of the discounted return G, For instance, with one-step bootstrapping, we have Ge s Tr+1 + +VV (s(+iii)) The critic objective is then Jv (v)))= = (r++1 + ++VVVst (8t+1; Φ) Vss;; Ø))² We minimize a weighted sum the policy objective and the critic objective (0, () - (0) Jv))
3.2 Dynamics modeling during pretraining
In addition policy learning, Language Dynamics Distillation lears dynamics model from unla- beled demonstrations to initialize and distill into the policy learner. Consider a set of demonstrations without labeled actions To {TI, 72, ... Tn. obtained by rolling out some policy (a,, st), where each demonstration т = [1,, ...... sT] consists of sequence of observations. We learn a dynamics model 0(s) ... 915 <) to predict the next observation 8 given the previous observations.
x
J₂(() = nf
Σ sim (0s,,, se: st:))
where sim differentiable similarity function between the predicted state o(s st) and the observed state St41, and are parameters of the dynamics model. In the environments we consider, sim is the cross-entropy loss across grid of symbols denoting entities present in the scene.
3.3 Dynamics distillation during policy learning
Fig shows the decomposition of the model into representation network freps a policy head fn. value head fy, and dynamics head fs. The three heads share parameters because their inputs

---

Nethack
Nethack ablations
2000
1500
retum
1000
500
LDD
random VAE
LDD
no-mig
reward
LDD dsll
no msg stats random
Inverse
LDD-init
0
step
1e9
step
1e9
Figure 2: NetHack Challenge comparisons (left) and ablations (right). LDD consistently outperforms other methods.
formed by the same representation network.
(st) fn (frep (8t: Prep): 0x) V(st) fv (Sop (st; frep) :8y
(3)
(4)
8(st) fs (frep (st 0rep) 0ss
(5)
In the first phase of dynamics modeling, we pretrain the model to predict future observations given demonstrations by optimizing Jg- We then store copy of the model as fixed teacher (st)) = fo (Trep sc: Grep) Let |X - Y denote the L2 distance between X and Y. During the second phase, addition to policy learning, we optimize a distillation objective Ja(Orep- 0,,0y) 0v) 1rrp (ss; (rep) - Seep (st: (uep) to avoid catastrophic forgetting of how to interpret language descrip- tions for dynamics modeling. This quantity the similarity between the feature representation produced by the fixed teacher (e.g. the pretrained dynamics model) and the feature representation produced by the model. Because is frozen, the parameters Örep are not included in the objective function Jg. The joint loss for Language Dynamics Distillation then
J [O 0x, 0v)
-J) + Jv + aaJa(0 Ox 0v)
To summarize, using unlabeled demonstrations with language descriptions, LDD learns dynamics model of the environment that grounds language descriptions to environment observations (Sec- tion B.2). This prior knowledge is then injected into reinforcement learning via initialization and distillation (Section 3.3
4 Experiments
We evaluate Language Dynamics Distillation on the Situated Interactive Language Grounding bench- mark (SILG) Zhong et al.) 2021). SILG consists of five different language grounding environments with diverse challenges term complexity of observation space, action space, language, and reasoning procedure. In all environments, situated agent observes symbolically (RTFM, Messenger, Nethack, Touchdown) or prose (ALFWorld) rendered visuals and interacts with the environment to follow some spsccfc language goals (e.g. what to do). In addition, the agent observes text manuals describing instance-agnoste environment rules (c.g. entity-rol, associations). The learning challenge leamn reading agent that generalizes new environments with different environment rules (e.g. new entity-team associations, new parts of the map). The five different environments are follows.
4.1 Environments
NetHack Küttler al. 2020 The agent must descend a procedurally generated dungeon. Its primarily challenge is in large state space and partial observability, the map remains obscured until

---

Touchdown
Touchdown ablations
0.15
win rate
00.10
0.05
LDD
random
LDD
random
reward
VAE
LDD-dsti
no-desc
0.00
inverse
LDD-init
frames
1a7
frames
1e7
Figure 3: Touchdown comparisons (left) and ablations (right). Methods that distill or reward shape (LDD, reward, LDD-init) outperform those that do not.
exploration. Observations include symbolic grid of entity IDs, in -a-aame message, and description of character stats. The agent chooses among a fixed set of actions such as movement, picking up/buying/selling items, and attacking. We evaluate not SILGNethack but the full NetHack challenge, a difficult game for humans with 155% expert win rate [Street 2013
Sym Touchdown Zhong et al.) 2021]: A symbolic version of Touchdown (Chen al.) 2018
where the agent navigates segmentation maps of Street View panoramas following long instructions. The primary challenge is reading long, natural instructions that describe photorealistic images. Evaluation is new navigation instructions. Observations include a grid of segmentation class IDs corresponding to discretized Google Street View panorama, synthetic captions of where objects are relative to the agent (e.g. to your right you see lot of road and some cars), and language navigation instructions. The agent selects from a list radial directions to proceed to the next panorama.
202 The agent navigates and manipulates objects inside kitchen which is described via textual descriptions. ALFWorld is challenging due to its large (>50) text action space that vary across scenes. Evaluation is on unseen instructions. Observations include a textual description of the scene and language goals (e.g. put clean sponge on the metal rack). The agent chooses among a variable set of language actions (e.g. open drawer I).
ALFWorld Shridhar
The agent interprets game manual and instruction acquire the correct items to fight the correct monsters. Its main challenge is in multi-step reasoning that combines world observations with texts describing multiple entities. Evaluation is set of manuals distinct from those in training; hence the agent cannot memorize training manuals and must learn to read correctly in order to generalize. Observations include symbolic grid containing names of entities present, manual describing high level game rules, agent inventory, and the language instruction. The agent chooses among fixed set of movements. We train and evaluate on the first curriculum stage.
RTFM
Zhong et al. 2020
The agent delivers message from source entity to target while avoiding an enemy. The entities are referred to in text by many names, which have no lexical overlap with their symbol ID, hence the core challenge is in mapping language entity references in text to observed symbolic entity IDs. Evaluations are on new entity-role assignments (e.g. who carries the message). Observations include symbolic grid containing symbol IDs of entities present, and manual entities and roles. The agent chooses among fixed set of movements. We train and evaluate the second curriculum stage.
Messenger HHanjie al. 2021
4.2 Method and Baselines
Reinforcement learning with language descriptions from scratch. We train a base tabula-rasa policy learner from random initialization. For NetHack, we train the base policy learner from [Küttler
al. 2020 using moolib [Mella et al. 2022 For RTFM, ALFWorld, and Touchdown, train the

---

ALFWor
ALFWorld
0.25
For Messenger, we train the win Late
0.15
0.10
0.05
LDD
random VAE
reward inverse
LDD
LDD-init
0.00
LDD-dsUl
random
frames
1e7
frames
1e7
Figure 4: ALFWorld comparisons (left) and ablations (right). LDD consistently outperforms other methods.
RTFM eval
RTFM ablation
1.0
0.8
vin rale
0.6
0.2
random
0.0
reward
VAE...
LDD
LDD-init
Inverse
LDD-dall
random
5
6
frames
1e7
frames
1e7
Figure 5: RTFM comparisons (left) and ablations (right). LDD consistently outperforms other methods. Because of the multi-step reasoning nature of RTFM solutions, partially complete strategies (e.g. able to do 2/3/4 cross-references) result in rep-wise gains in win-rates. Strategies different levels of completion result in larger variances when averaged.
SIR model from Zhong et al. 2021 using Torchbeast Küttler et al.
2019 Espeholt al., 2018).
For Messenger, train the EMMA model from Hanjie et al. 2222 using PPO [Schulman et al.)
2017]. For NetHack, we train the base policy learner from Küttee et al. 2020 using moolib MMela
et al. 2022 For RTFM, ALFWorld, and Touchdown, we train the SIR model from [Zhong et al.
2021 using Torchbeast Kütttler et al. 2019 Espeholt et al. 2018
EMMA model from Hanjie et al. 2021 using PPO [Schulman al. 2017
Pretraining representations via a variational autoencoder. We pretrain a variational autoencoder (VAB), common approach for representation learning, that predicts the intermediate representation just before the policy head [Kingma and Welling 2013]. This VAE has the same architecture as the policy learner, and used initialize the policy learner. The training procedure for the VAE is as described in Ha and Schmidhuber 2018).
Language Dynamies Distillation (LDD) We train LDD variants of the baseline policy learners for each environment, where we pretrain the model to perform dynamics modeling unannotated demonstrations. For NetHack, we use 100k screen-recordings (where actions are annotated and cannot be trivially reverse engineered due to ambiguity in observations) of human- mamman-playtrouughss from the alt.org NetHack public server. For Touchdown, we use unannotated demonstrations by human players. For ALFWorld, we use trajectories obtained from A planner with full state and goal knowledge, with actions removed. For RTFM and Messenger, we train expert trajectories until convergence, and sample 10k rollouts from the experts from which we remove action labels.

---

Messenger-SS
Messenger-SZ ablations
0.4
-0.5
-0.6
reward
-077
-088
0.9
LDD
VAE
-1.0
reward inveree
random
LDD
LDD-init
LDD-dstl
random
-1.1
step
1eB
step
1e6
Figure 6: Messenger comparisons (left) and ablations (right). Methods that pretrain to initialize (LDD, inverse, LDD-distill) outperform those that do not.
Reward shaping with expert. Methods such Merel et encouraging the agent to produce states similar to demonstrator. To compare LDD to this idea, we use the dynamics model to predict the next observation under expert policy. The difference (e.g. accuracy in symbol prediction across grid) between the predicted observation and the actual observation after taking the action proposed by the agent is used as a penalty (e.g. negative auxiliary reward). This method is listing reward in experiment figures and uses the same unlabeled demonstrations as LDD.
2017 reward shape with an expert by
Inverse reinforcement learning. Another class of methods learn inverse dynamics model with which infer actions in unlabeled demonstrations, then learn to imitate the pseudo-labeds labeled demonstra- tions [Hanna and Stone 2017 Torabi et al.) 2018]. To compare this method, we first train the base policy learner for 10k episodes, then collect 10k rollouts to train an inverse dynamics model that predicts current action given current and future observations. This inverse model is used annotate the original unlabeled demonstrations for imitation learning (Torabi et al.] [018)_ Because these imitation policies do not generalize novel environments and goals found SILG evaluation, additionally fine-tune them via RL similar to Guo et al. 2019). This method is listing as inverse in experiment figures. In addition the data used by reward and LDD, inverse uses additional rollouts to train the inverse dynamics model.
4.3 Results and ablations
LDD consistently improves performance across environments. We evaluate on held-out envi- ronments across random seeds for NetHack in Fig Touchdown in Fig 3 ALFWorld in Fig 4 RTFM in Fig 5 and Messenger in Fig 6 LDD obtains top performance compared to tabula-ras policy learning with language descriptions, VAE pretraining, reward shaping using the dynam- ics model, and inverse reinforeement learning. This is consistent across challenges in multi-step reasoning (RTFM), language-entity generalization (Messenger), large language action spaces (ALF- World), large procedurally generated states (NetHack), and long natural language instructions with complex visual scenes (Touchdown). We also ablate LDD by removing the initialization step (LDD-init) the distillation step (LDD-distill). On Messenger, methods that pretrain to ini- tialize (LDD, inverse, LDD (DD-distill) outperform those that do not (reward, LDD-init). On Touchdown, methods that distill reward shape (LDD, reward, LDD-init) outperform those that do not. Learning curves for each method across environments are shown in Appendix section G LDD converges faster and higher win rate than other methods, with the exception of Touchdown, where it achieves lower training but higher evaluation win-rate.
Adding language descriptions improves performance. What is the role of language descrip- tions in pretraining and subsequent policy learning? To answer this question, we ablate language descriptions by removing them from the environments in NetHack and SymToochhenn Touchdown (the other envi- ronments are solvable without descriptions because they describe the objective). no-msg removes NetHack messages describing events near the agent (c.g. kitten attacks the bat!). no-mag, stats

---

100
80
60
800
20
random expert
nethack
touchdown
alfworld
rtfm
messenger
Figure 7: Dynamics model frame prediction accuracy using unlabeled expert demonstrations vs. rollouts from random policies. Training using expert demonstrations result in accurate dynamics models, especially on complex environments that are difficult to explore via random policies.
additionally removes character state descriptions (e-g. health, achievements, dungeon level). no no-desc removes Touchdown captions that describe objects locations scene relative to the agent (e.g. on your left, there are many building, some people, and few cars). These variants differ only in the observation space used in dynamics modeling. In fine-tuning, they receive the same observation space with lan- guage descriptions. For both NetHack (Fig 2 and Touchdown (Fig S removal of in-game messages, character state descriptions, and synthetic captions degrade performance. Appendix Fig 12 further shows that adding language descriptions results in dynamics models with higher accuracy. This suggests that modeling language descriptions in the observation space result in better initialization and distillation, which improve subsequent policy learning.
Expert demonstrations cover late-stage strategy that result in asymptotic gains. In complex environments, experts demonstrate late stage strategies diffieult to explore via random sampling. Because these stages are rarely reached, accurate dynamics models are especially helpful in providing signals when rewards are sparse. Take NetHack an example: unlike experts, non-expert policies never proceed to the deeper dungeons of the game. A dynamics model trained on non-expert rollouts therefore struggles to generalize to unseen deeper dungeons. As the agent learns and descends deeper in the dungeon, dynamics model trained on non-expert demonstrations results in less distillation gains. Fig 7 compares dynamies modeling from observations using expert demonstrations vs. using rollouts from a random policy. The evaluation is on held-out set of expert demonstrations. Across environments, training on expert demonstrations outperforms training on non-expert demonstrations. This effect is less apparent environments where random policies can (eventually) discover most of the state space (e.g, RTFM, Messenger) more apparent on partially observed environments where only strategic expert policies encounter rare states indicative of success (c.g. long-term planning in NetHack and Touchdown, choosing from large language action space in ALFWorld).
Conclusion
While recent work showed that augmentation with language descriptions result in better policies, learning how to ground language descriptions to observations is difficult through naive RI. with sparse, delayed rewards. We proposed Language Dynamics Distillation, which pretrains dynamics model using cheaply-obtained unlabeled demonstrations with language descriptions to initialize and distill into the policy learner. On five tasks with language descriptions, LDD improved sample efficiency and resulted in better policies than RL from scratch, inverse RL, and expert reward shaping. In addition, the benefit from initialization and distillation differ on an environment basis, but are complemenarry across environments. Moreover, language descriptions improved initialization and distillation gains in policy learning. Finally, learning model dynamics with expert demonstrations was more effective than with non-expert rollouts. A promising direction for future research is studying whether dynamics modeling with language descriptions similarly effective in robotic control where naive RL. can be prohibitively expensive, but unlabeled demonstrations with synthetic captions are cheap obtain.

---

Acknowledgments and Disclosure of Funding
We are grateful the anonymous reviewers for their helpful comments and suggestions. Victor is supported in part by the ARO (AROW91 and by the Apple AI/ML fellowship.
References
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton den Hengel. Vision-and-language navigation: Interpreting visually- grounded navigation instructions in real environments. In CVPR, 2018.
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In ICLR, 2019.
Howard Chen, Alane Suhr. Dipendra Kumar Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, 2018.
Ashley D Edwards, Himanshu Sahni, Yannick Schroecker, and Charles L Isbell. Imitating Latent Policies from Observation. In ICML, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. IMPALA: Scalable distributed deep-rl with importance weighted actor-learner architectures. ICML, 2018.
Xiaoxiao Guo, Shiyu Chang, Mo Yu, Gerald Tesauro, and Murray Campbell. Hybrid reinforcement learning with expert state sequences. In AAAI, 2019.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In NeurlPS, 2018.
Austin W. Hanjie, Victor Zhong, and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. In ICML, 2021.
Josiah P. Hanna and Peter Stone. Grounded Action Transformation for Robot Learning in Simulation. In AAAI, 2017.
He He. Derek Chen, Anusha Balakrishnan, and Percy Liang. Decoupling strategy and generation in negotiation dialogues. EMNLP, 2018
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR, 2017.
Daiki Kimura, Subhajit Chaudhury, Ryuki Tachibana, and Sakyasingha Dasgupta. Internal Model from Observations for Reward Shaping. ALA workshop at ICML, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv, 1312.6114, 2013.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-Acroo Across-Room: Multilingual vision-andd-laguuae navigation with dense spatiotemporal grounding. In EMNLP, 2020.
Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. TorchBeast: A PyTorch Platform for Distributed RL. arXiv, 1910.03552. 2019.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack learning environment. In NewrIPS, 2020.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In ICLR, 2018.
Vegard Mella, Eric Hambro, Danielle Rothermel, and Heinrich Küttler. moolib: A platform for distributed RL. http::/gghhbcoommmccccooorrr///s 2022.
10

---

Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv, 1707.02201, 2017.
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. In EMNLP, 2015.
Roberta Raileanu and Tim Rocktäschel. RIDE: Rewarding Impact-Driven Exploration for Procedurally-GGeeeeee Environments. In ICLR, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, 1707.06347, 2017.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In CVPR, 2020.
Mohit Shridhar, Xingdi Yuan, Alee-AAee Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In ICLR, 2021.
Bradly C. Stadie, Pieter Abbeel, and Iya Sutskever. 1703.01703, 2017.
Third-person imitation learning. arXiv,
Chris Street. Observations on expert play in nethack. https///oohhaay...eee/ettaaaacea htm 2013.
Alexander Strehl and Michael L.. Littman. An analysis of model-based interval estimation for markov decision processes. JCSS, 2008
Allison C. Tam, Neil C. Rabinowitz, Andrew K. Lampinen, Nicholas A. Roy, Stephamie C. Y. Chan, Strouse, Jane X. Wang, Andrea Banino, and Felix Hill. Semantic exploration from language abstractions and pretrained representations, 2022.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In IJCAI, 2018.
Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. In NeurlPS, 2019.
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. RTFM: Generalising to new environment dynamics via reading. In ICLR, 2020.
Victor Zhong, Austin W Hanjie, Sida I Wang, Karthik Narasimhan, and Luke Zettlemoyer. SILG: The multi-environment symbolic interactive language grounding benchmark. In NeurlPS, 2021.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? Yes [Yes]
(b) Did you describe the limitations of your work? [Yes] See section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes] See section C
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See section D
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See section E
3. If you using existing assets (e.g., code, data, models) curating/releusing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See section
(c) Did you include any new assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [Yes] See section F
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] See section
12


Reviews:
[{"review_id": "u5Ww-BUGjn_", "reviewer": "Reviewer_ZHyY", "summary": "Sparse reward environments can be challenging for language-conditioned policies as credit assignment to the language description can prove difficult. The authors pretrain on a dataset of language-conditioned experts’ observations and distill this into the RL policy to demonstrate strong performance gains on a set of language-guided environments. I recommend accept for this paper, but would like to see a few things (mainly regarding ablations) addressed.", "questions": "None", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n**Writing:** The motivation/method are presented clearly.\n\n**Novelty:** The method is seemingly novel yet simple.\n\n**Results:** LDD seems to work pretty well and provide strong improvements over the baselines in the paper across the some of the environments.\n\n## Weaknesses\n\n**Writing:** The last paragraph of the intro is unnecessarily long and reads more like an experiments analysis section. The intro would be more readable by greatly shortening this paragraph and leaving the detailed experiments results to the experiments section.\n\n**Missing Ablation:** While I am pretty sure this baseline would be outperformed by LDD, the authors should test LDD trained without language guidance (just predicting next states). I see appendix Figure H does this partly, but this only compares with/without language inputs for dynamics modeling accuracy, not downstream RL performance.\n\nAdditionally, the authors should also include at least one other baseline that uses the same dataset in some way. For example, perhaps an LDD type of model that is pretrained with contrastive learning or some other representation learning objective (predict whether  observations are from the same trajectory given the language description), to verify that the dynamics loss is the best commonly used objective to apply here.\n\n**Minor issues:** \n\n- L220: “expert policy instead of for”\n- Hyperparams: It’s useful for papers to at least have a small table/description of how hyperparameters were selected for all baselines and the main method, and how much tuning was done.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fiQYxzkEttj", "reviewer": "Reviewer_xVZE", "summary": "This paper presents a new method, Language Dynamics Distillation(LDD), which pre-trains the model to predict environment dynamics with given demonstrations and language descriptions, then fine-tunes the final policy via reinforcement learning on pre-trained representations. The main contribution claimed by the authors is that this method can learn the RL policy more efficiently. It is especially effective when the reward is sparse or delayed, and it is a realistic setting because it assumes a cheap additional expert demonstration (ex. video) without action labeling.", "questions": "- I think the authors should have clearly defined and explained the problem setting (ex. about language description)  in Chapter 3. Does the defined s_t include a language description? If not, a detailed explanation of exactly how the language description is used seems necessary.\n- Among the baseline algorithms, how was ‘reward shaping with expert’ (line 218-219) applied specifically?\n- Can the proposed method be used with general intrinsic reward-related methods (ex. RND, ICM, …)? If so, I wonder if it still works effectively when applied with those methods.\n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- It considers the realistic setting that assumes only cheap additional expert demonstration without action labeling.\n\nWeaknesses\n- It seems that the paper does not provide a clear explanation of the problem setting and the details of the proposed algorithm.\n- It seems that the baseline algorithms provided in the experiment are not sufficient. It would be better if it was compared with several algorithms which are proposed to address the settings of sparse and delayed rewards. (ex. RND [1], ICM [2], …)\n\n[1] Yuri Burda et al, Exploration by random network distillation, ICLR 2019\n\n[2] Deepak Pathak et al, Curiosity-driven Exploration by Self-supervised Prediction, ICML 2017\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IgzpDcb5aYR", "reviewer": "Reviewer_W5e2", "summary": "The paper proposes to learn a language-conditioned dynamics model (without action labels), and shows that the representation learned by this dynamics model can improve downstream policy training, when combined with necessary techniques such as distillation. The method is evaluated on the SILG benchmark, which includes several distinct environments. Compared to baselines such as pure RL, inverse RL, reward-shaped RL, the proposed method is shown to outperform or on par with the best of all baselines across the benchmark. It is also ablated across several variants to show dynamics modeling with language outperforms without language, and that using expert trajectories for dynamics modeling is important.\n\n--------------\nPost Author Response Update\n\nThe authors have addressed my major concern for the paper. I'm updating my review score to 6.", "questions": "Please refer to the comments in the weakness section.\n\nTo summarize:\n\n1. It is suggested to better contextualize the work around representation learning for RL, while advocating the importance of language modeling from this perspective.\n2. In this regard, any comparison to a well-known or state-of-the-art method in “representation for RL” could be very useful to evaluate the significance of the contribution of this work.", "limitations": "The authors did not adequately address the limitations of this work; it is only stated that the limitation is that the environments are simulated. This is an insignificant limitation and not useful to evaluate the significance of the contribution of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper text is written with clarity.\n2. The method appears to be novel for the reviewer.\n3. The paper investigates important question in the field: how can language modeling help decision-making problems?\n4. The evaluations include several distinct environments and tasks.\n\nWeaknesses:\n\nMy concern is mostly around the significance of the paper. While it’s shown to be better than various baselines, the paper is not contextualized well enough for various prior works in representation learning for RL. While many prior works in this domain may not involve language modeling, given the nature of the proposed method, it should be discussed how it differs from other representation learning methods for RL and positions the paper from this perspective. Moreover, while the paper shows that the proposed method outperforms various baselines, it is susceptible that the improvement mostly comes from the good representation modeling. Therefore, this can only be a meaningful contribution if the paper could show comparisons to prior works in representation learning for RL. For example, what if the expert trajectories are used to train a representation module that is not a dynamics model, using some objectives? Would this representation work well in these tasks? If not, is it because they don’t model language?\n\nMinor improvement include:\n\n1. While the text is relatively clear in the paper, Figure 1 is poorly made, with blurry environment pictures and unclear method visualization. Potential improvement, for instance, can be separating the “traditional training loop” from the proposed method.\n2. Fig 6b seems to be missing several learning curves.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "81utqpezyf3", "reviewer": "Reviewer_cNeZ", "summary": "This paper proposes to ground policy representations on representation learned from language conditioned dynamic modeling. The paper argues that this is a cheap approach to obtain langauge grounding from observations, and illustrates across a set of different textual domains how such an approach can lead to improved RL performance.", "questions": "See above", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is relatively clear to read, and the evaluation makes sense to me\n\nWeaknesses:\n\nThe underlying proposed method seems a bit trivial to me -- the author simply proposed to finetune on a dynamics model.\n\nThe benefit of using expert data to train the dynamics model compared random data in Figure 7 seems to be minimal.\n\nThe underlying rewards curves of different models have very high variance. For example see the ablations in Figure 5.\n\nThe performance of the proposed method also appears to similar to other methods (see Figure 3)\n\nIntuitively -- why does such a dynamic modeling objective lead to grounded language representations? Can the authors illustrate this more convincingly with more qualitative visualizations or quantitative grounding results? Also illustrating this qualitative difference with using random frames would be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept