Venue: NAACL 2018

# Learning with Latent Language

Jacob Andreas Dan Klein Sergey Levine

Computer Science Division

University of California, Berkeley

{jda,klein,svlevine}@eecs.berkeley.edu

###### Abstract

The named concepts and compositional operators present in natural language provide a rich source of information about the abstractions humans use to navigate the world. Can this linguistic background knowledge improve the generality and efficiency of learned classifiers and control policies? This paper aims to show that using the space of natural language strings as a _parameter_ space is an effective way to capture natural task structure. In a pretraining phase, we learn a language interpretation model that transforms inputs (e.g. images) into outputs (e.g. labels) given natural language descriptions. To learn a new concept (e.g. a classifier), we search directly in the space of descriptions to minimize the interpreter's loss on training examples. Crucially, our models do not require language data to learn these concepts: language is used only in pretraining to impose structure on subsequent learning. Results on image classification, text editing, and reinforcement learning show that, in all settings, models with a linguistic parameterization outperform those without.1

Footnote 1: Code and data are available at [https://github.com/jacobandreas/13](https://github.com/jacobandreas/13).

## 1 Introduction

The structure of natural language reflects the structure of the world. For example, the fact that it is easy for humans to communicate the concept _left of the circle_ but comparatively difficult to communicate _mean saturation of the first five pixels in the third column_ reveals something about the abstractions we find useful for interpreting and navigating our environment Gopnik and Meltzoff (1987). In machine learning, efficient automatic discovery of reusable abstract structure remains a major challenge. This paper investigates whether background knowledge from language can provide a useful scaffold for acquiring it. We specifically propose to use language as a latent _parameter space_ for few-shot learning problems of all kinds, including classification, transduction and policy search. We aim to show that this linguistic parameterization produces models that are both more accurate and more interpretable than direct approaches to few-shot learning.

Like many recent frameworks for multitask- and meta-learning, our approach consists of three phases: a pretraining phase, a concept-learning phase, and an evaluation phase. Here, the product of pretraining is a language interpretation model that maps from descriptions to predictors (e.g. image classifiers or reinforcement learners). Our thesis is that language learning is a powerful, general-purpose kind of pretraining, even for tasks that do not directly involve language.

New concepts are learned by searching directly in the space of natural language strings to mini

Figure 1: Example of our approach on a binary image classification task. We assume access to a pretrained language interpretation model that outputs the probability that an image matches a given description. To learn a new visual concept, we search in the space of natural language descriptions to maximize the interpretation model’s score (top). The chosen description can be used with the interpretation model to classify new images (bottom). Figures best viewed in color.



mize the loss incurred by the language interpretation model (Figure 1). Especially on tasks that require the learner to model high-level compositional structure shared by training examples, natural language hypotheses serve a threefold purpose: they make it easier to discover these compositional concepts, harder to overfit to few examples, and easier for humans to understand inferred patterns.

Our approach can be implemented using a standard kit of neural components, and is simple and general. In a variety of settings, we find that the structure imposed by a natural-language parameterization is helpful for efficient learning and exploration. The approach outperforms both multitask- and meta-learning approaches that map directly from training examples to outputs by way of a real-valued parameterization, as well as approaches that make use of natural language annotations as an additional supervisory signal rather than an explicit latent parameter. The natural language concept descriptions inferred by our approach often agree with human annotations when they are correct, and provide an interpretable debugging signal when incorrect. In short, by equipping models with the ability to "think out loud" when learning, they become both more comprehensible and more accurate.

## 2 Background

Suppose we wish to solve an image classification problem like the one shown in Figure 2b-c, mapping from images \(x\) to binary labels \(y\). One straightforward way to do this is to solve a learning problem of the following form:

\[\operatorname*{arg\,min}_{\eta\in\mathbf{H}}\quad\sum_{(x,\,y)}L(f(x;\,\eta), \,y)~{}, \tag{1}\]

where \(L\) is a loss function and \(f\) is a richly-parameterized class of models (e.g. convolutional networks) indexed by \(\eta\) (e.g. weight matrices) that map from images to labels. Given a new image \(x^{\prime}\), \(f(x^{\prime};\,\eta)\) can be used to predict its label.

In the present work, we are particularly interested in _few-shot_ learning problems where the number of \((x,\,y)\) pairs is small--on the order of five or ten examples. Under these conditions, directly solving Equation 1 is a risky proposition--any model class powerful enough to capture the true relation between inputs and outputs is also likely to overfit. For few-shot learning to be successful, extra structure must be supplied to the learner. Existing approaches obtain this structure by either carefully structuring the hypothesis space or providing the learner with alternative training data. The approach we present in this paper combines elements of both, so we begin with a review of existing work.

(Inductive) **program synthesis** approaches (e.g. Gulwani, 2011) reduce the effective size of the hypothesis class \(\mathbf{H}\) by moving the optimization problem out of the continuous space of weight vectors and into a discrete space of formal program descriptors (e.g. regular expressions or Prolog queries). Domain-specific structure like version space algebras Lau et al. (2003) or type systems Kitzelmann and Schmid (2006) can be brought to bear on the search problem, and the bias inherent in the syntax of the formal language provides a strong prior. But while program synthesis techniques are powerful, they are also limited in their application: a human designer must hand-engineer the computational primitives necessary to compactly describe every learnable hypothesis. While reasonable for some applications (like string editing), this is challenging or impossible for others (like computer vision).

An alternative class of **multitask learning** approaches Caruana (1998) import the relevant structure from other learning problems rather than defining it manually (Figure 2a, top). Since we may not know _a priori_ what set of learning problems we ultimately wish to evaluate on, it is useful to think of learning as taking places in three phases:

Figure 2: Formulation of the learning problem. Ultimately, we care about our model’s ability to learn a concept from a small number of training examples (b) and successfully generalize it to held-out data (c). In this paper, concept learning is supported by a language learning phase (a) that makes use of natural language annotations on other learning problems. These annotations are not provided for the real target task in (b–c).

 

1. a **pretraining** (or "meta-training") phase that makes use of various different datasets with examples \(\{(x_{1}^{(fi)},y_{1}^{(fi)}),\ldots,(x_{n}^{( fi)},y_{n}^{(i)})\}\) (Figure 2a)
2. a **concept-learning** phase in which the pretrained model is adapted to fit data \(\{(x_{1}^{(c)},y_{1}^{(c)}),\ldots,(x_{n}^{(c)},y_{n}^{(c)})\}\) for a specific new task (Figure 2b)
3. an **evaluation** phase in which the learned concept is applied to a new input \(x^{(e)}\) to predict \(y^{(e)}\) (Figure 2c)

In these approaches, learning operates over two collections of parameters: shared parameters \(\eta\) and task-specific parameters \(\theta\). In pretraining, multitask approaches find:

\[\operatorname*{arg\,min}_{\eta\in\mathbb{R}^{a},~{}\theta^{(\ell i)}\in \mathbb{R}^{b}}~{}~{}\sum_{i,j}L\big{(}f(x_{j}^{(\ell i)};\,\eta,\,\theta^{( \ell i)}),\,y_{j}^{(\ell i)}\big{)}\;. \tag{2}\]

At concept learning time, they solve for:

\[\operatorname*{arg\,min}_{\theta^{(e)}\in\mathbb{R}^{b}}~{}~{}\sum_{j}L\big{(} f(x_{j}^{(c)};\,\eta,\,\theta^{(c)}),\,y_{j}^{(c)}\big{)} \tag{3}\]

on the new dataset, then make predictions for new inputs using \(f(x^{(e)};\,\eta,\,\theta^{(c)})\).

Closely related **meta-learning** approaches (e.g. Schmidhuber, 1987; Santoro et al., 2016; Vinyals et al., 2016) make use of the same data, but collapse the inner optimization over \(\theta^{(c)}\) and prediction of \(y^{(e)}\) into a single learned model.

## 3 Learning with Language

In this work, we are interested in developing a learning method that enjoys the benefits of both approaches. In particular, we seek an intermediate language of task representations that, like in program synthesis, is both expressive and compact, but like in multitask approaches is learnable directly from training data without domain engineering. We propose to use natural language as this intermediate representation. We call our approach **learning with latent language** (\(\mathrm{L}^{3}\)).

Natural language shares many structural advantages with the formal languages used in synthesis approaches: it is discrete, has a rich set of compositional operators, and comes equipped with a natural description length prior. But it also has a considerably more flexible semantics. And crucially, plentiful annotated data exists for _learning_ this semantics: we cannot hand-write a computer program to recognize a _small dog_, but we can learn how to do it from image captions. More basically, the set of primitive operators available in language provides a strong prior about the kinds of abstractions that are useful for natural learning problems.

Concretely, we replace the pretraining phase above with a **language-learning** phase. We assume that at language-learning time we have access to natural-language **descriptions**\(w^{(\ell i)}\) (Figure 2a, bottom). We use these \(w\) as _parameters_, in place of the task-specific parameters \(\theta\)--that is, we learn a language **interpretation** model \(f(x;\,\eta,\ w)\) that uses shared parameters \(\eta\) to turn a description \(w\) into a function from inputs to outputs. For the example in Figure 2, \(f\) might be an image rating model (Socher et al., 2014) that outputs a scalar judgment \(y\) of how well an image \(x\) matches a caption \(w\).

Because these natural language parameters are observed at language-learning time, we need only learn the real-valued shared parameters \(\eta\) used for their interpretation (e.g. the weights of a neural network that implements the image rating model):

\[\operatorname*{arg\,min}_{\eta\in\mathbb{R}^{a}}~{}~{}\sum_{i,j}L\big{(}f(x_{ j}^{(\ell i)};\,\eta,\,w^{(\ell i)}),\,y_{j}^{(\ell i)}\big{)}\;. \tag{4}\]

At concept-learning time, conversely, we solve only the part of the optimization problem over natural language strings:

\[\operatorname*{arg\,min}_{w^{(c)}\in\Sigma^{*}}~{}~{}\sum_{j}L\big{(}f(x_{j}^ {(c)};\,\eta,\,w^{(c)}),\,y_{j}^{(c)}\big{)}\;. \tag{5}\]

This last step presents something of a challenge. When solving the corresponding optimization problem, synthesis techniques can exploit the algebraic structure of the formal language, while end-to-end learning approaches take advantage of differentiability. Here we can't do either--the language of strings is discrete, and any structure in the interpretation function is wrapped up inside the black box of \(f\). Inspired by related techniques aimed at making synthesis more efficient (Devlin et al., 2017), we use learning to help us develop an effective optimization procedure for natural language parameters.

In particular, we simply use the language-learning datasets, consisting of pairs \((x_{j}^{(\ell i)},y_{j}^{(\ell i)})\) and descriptions \(w_{i}\), to fit a reverse **proposal** model, estimating:

\[\operatorname*{arg\,max}_{\lambda}\sum_{i}\log q(w_{i}|x_{1}^{(\ell i)},y_{1}^ {(\ell i)},\ldots,x_{n}^{(\ell i)},y_{n}^{(\ell i)};\lambda)\] 

where \(q\) provides a (suitably normalized) approximation to the distribution of descriptions given task data. In the running example, this proposal distribution is essentially an image captioning model Donahue et al. (2015). By sampling from \(q\), we expect to obtain candidate descriptions that are likely to obtain small loss. But our ultimate inference criterion is still the true model \(f\): at evaluation time we perform the minimization in Equation 5 by drawing a fixed number of samples, selecting the hypothesis \(w^{(c)}\) that obtains the lowest loss, and using \(f(x^{(e)};\,\eta,\,w^{(c)})\) to make predictions.

What we have described so far is a generic procedure for equipping collections of related learning problems with a natural language hypothesis space. In Sections 4 and 5, we describe how this procedure can be turned into a concrete algorithm for supervised classification and sequence prediction. In Section 6, we describe how to extend these techniques to reinforcement learning.

## 4 Few-shot Classification

We begin by investigating whether natural language can be used to support high-dimensional few-shot classification. Our focus is on visual reasoning tasks like the one shown in Figure 3. In these problems, the learner is presented with four images, all positive examples of some visual concept like _a blue shape near a yellow triangle_, and must decide whether a fifth, held-out image matches the same concept. These kinds of reasoning problems have been well-studied in visual question answering settings Johnson et al. (2017); Suhr et al. (2017). Our version of the problem, where the input and output feature no text data, but an explanation must be inferred, is similar to the visual reasoning problems proposed by Raven (1936) and Bongard (1968).

To apply the recipe in Section 2, we need to specify an implementation of the interpretation model \(f\) and the proposal model \(q\). We begin by computing representations of input images \(x\). We start with a pre-trained 16-layer VGGNet Simonyan and Zisserman (2014). Because spatial information is important for these tasks, we extract a feature representation from the final convolutional layer of the network. This initial featurization is passed through two fully-connected layers to form a final image representation, as follows:

We define interpretation and proposal models:2

Footnote 2: Suppressing shared parameters \(\eta\) and \(\lambda\) for clarity.

\[f(x;w)=\sigma\big{(}rnn\text{-encode}(w)^{\top}rep(x)\big{)}\] \[q(w\mid\{x_{j}\})=rnn\text{-decode}\big{(}w\mid\frac{1}{n}\sum _{j}rep(x_{j})\big{)}\]

The interpretation model \(f\) outputs the probability that \(x\) is assigned a positive class label, and is trained to maximize log-likelihood. Because only positive examples are provided in each language learning set, the proposal model \(q\) can be defined in terms of inputs alone. Details regarding training hyperparameters, RNN implementations, etc. may be found in Appendix A.

Our evaluation aims to answer two questions. First, does the addition of language to the learning process provide any benefit over ordinary multitask or meta-learning? Second, is it specifically better to use language as a hypothesis space for concept learning rather than just an additional signal for pretraining? We use several baselines to answer these questions:

1. _Multitask_: a multitask baseline in which the definition of \(f\) above is replaced by \(\sigma(\theta_{i}^{\top}rep(x))\) for task-specific parameters \(\theta_{i}\) that are optimized during both pretraining and concept-learning.
2. _Meta_: a meta-learning baseline in which \(f\) is defined by \(\sigma([\frac{1}{n}\sum_{j}rep(x_{j})]^{\top}rep(x))\).3 Footnote 3: Many state-of-the-art approaches to meta-learning for classification (e.g. Snell et al., 2017) are not well-defined for possibly-overlapping evaluation classes with only positive examples provided. Here we have attempted to provide a robust implementation that is as close as possible to the other systems under evaluation.

Figure 3: The few-shot image classification task. Learners are shown four positive examples of a visual concept (left) and must determine whether a fifth image matches the pattern (right). Natural language annotations are provided during language learning but must be inferred for concept learning.

 

3. _Meta+Joint:_ as in _Meta_, but the pretraining objective includes an additional term for predicting \(q\) (discarded for concept learning).

We report results on a dataset derived from the ShapeWorld corpus of Kuhnle and Copestake (2017). In this dataset the held-out image matches the target concept 50% of the time. In the validation and test folds, half of learning problems feature a concept that also appears in the language learning set (but with different exemplar images), while the other half feature both new images and a new concept. Images contain two or three distractor shapes unrelated to the objects that define the target concept. Captions in this dataset were generated from DMRS representations using an HPS grammar Copestake et al. (2016). (Our remaining experiments use human annotators.) The dataset contains a total of 9000 pretraining tasks and 1000 of each validation and test tasks. More dataset statistics are provided in Appendix B.

Results are shown in Table 1. It can be seen that L\({}^{3}\) provides consistent improvements over the baselines, and that these improvements are present both when identifying new instances of previously-learned concepts and when discovering new ones. Some example model predictions are shown in Figure 4. The model often succeeds in making correct predictions, even though its inferred descriptions rarely match the ground truth. Sometimes this is because of inherent ambiguity in the description language (Figure 4a), and sometimes because the model is able to rule out candidates on the basis of partial captions alone (Figure 4b, where it is sufficient to recognize that the target concept involves a _circle_). More examples are provided in Appendix C.

## 5 Programming by Demonstration

Next we explore whether the same technique can be applied to tasks that involve more than binary similarity judgments. We focus on structured prediction: specifically a family of string processing tasks. In these tasks, the model is presented with examples of five strings transformed according to some rule; it must then apply an appropriate transformation to a sixth (Figure 5). Learning proceeds as in the previous section, with:

\[\texttt{rep}(x,y)=\texttt{rnn-encode}([x,y])\] \[f(y\mid x;w)=\] \[q(w\mid\left\{(x_{j},\,y_{j})\right\})=\] \[\texttt{rnn-decode}\big{(}w\mid\frac{1}{n}\sum_{j\in\texttt{ rep}(x_{j},y_{j})}\big{)}\]

Baselines are analogous to those for classification.

While string editing tasks of the kind shown in Figure 5 are popular in both the programming by demonstration literature Singh and Gulwani (2012) and the semantic parsing literature Kushman and Barzilay (2013), we are unaware of any datasets that support both learning paradigms at the same time. We have thus created a new dataset of string editing tasks by (1) sampling random regular transducers, (2) applying these transducers to collections of dictionary words, and (3) showing the collected examples to Mechanical Turk users

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Model & Val (old) & Val (new) & Val & Test \\ \hline Random & 50 & 50 & 50 & 50 \\ Multitask & 64 & 49 & 57 & 59 \\ Meta & 63 & 62 & 62 & 64 \\ Meta+Joint & 63 & 69 & **66** & 64 \\ L\({}^{3}\) (ours) & 70 & 72 & **71** & **70** \\ L\({}^{3}\) (oracle) & 77 & 80 & 79 & 78 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Evaluation on image classification. _Val (old)_ and _Val (new)_ denote subsets of the validation set that contain respectively previously-used and novel visual concepts. L\({}^{3}\) consistently outperforms alternative learning methods based on multitask learning, meta-learning, and meta-learning jointly trained to predict descriptions (_Meta+Joint_). The last row shows results when the model is given a ground-truth concept description rather than having to infer it from examples.

Figure 4: Example predictions for image classification. The model achieves high accuracy even though predicted descriptions rarely match the ground truth. High-level structure like the presence of certain shapes or spatial relations is consistently recovered.

 

and asking them to provide a natural language explanation with their best guess about the underlying rule. The dataset thus features both multi-example learning problems, as well as structured and unstructured annotations for each target concept. There are 3000 tasks for language learning and 500 tasks for each of validation and testing (Appendix B). Annotations are included in the code release for this paper.

Results are shown in Table 2. In these experiments, all models that use descriptions have been trained on the natural language supplied by human annotators. While we did find that the Meta+Joint model converges considerably faster than all the others, its final performance is somewhat lower than the baseline Meta model. As before, L\({}^{3}\) outperforms alternative approaches for learning directly from examples with or without descriptions.

Because all of the transduction rules in this dataset were generated from known formal descriptors, these tasks provide an opportunity to perform additional analysis comparing natural language to more structured forms of annotation (since we have access to ground-truth regular expressions) and more conventional synthesis-based methods (since we have access to a ground-truth regular expression execution engine). We additionally investigate the effect of the number of samples drawn from the proposal model. These results are shown in Table 3.

A few interesting facts stand out. Under the ordinary evaluation condition (with no ground-truth annotations provided), language-learning with natural language data is actually better than language-learning with regular expressions. This might be because the extra diversity helps the model determine the relevant axes of variation and avoid overfitting to individual strings. Allowing the model to do its own inference is also better than providing ground-truth natural language descriptions, suggesting that it is actually better at generalizing from the relevant concepts than our human annotators (who occasionally write things like _I have no idea_ for the inferred rule). Unsurprisingly, with ground truth REs (which unlike the human data are always correct) we can do better than any of the models that require inference. Coupling our inference procedure with an oracle RE evaluator, we essentially recover the synthesis-based approach of Devlin et al. (2017). Our findings are consistent with theirs: when an exact execution engine is available, there is no reason not to use it. But we can get almost 90% of the way there

Figure 5: Example string editing task. Learners are presented with five examples of strings transformed according to some rule (left), and must apply an appropriate transformation to a sixth string (right). Language-learning annotations (center) may take the form of either natural language or regular expressions.

Figure 6: Example predictions for string editing.

 

with an execution model learned from scratch. Examples of model behavior are shown in Figure 6; more may be found in Appendix D.

## 6 Policy Search

The previous two sections examined supervised settings where the learning signal comes from few examples but is readily accessible. In this section, we move to a set of reinforcement learning problems, where the learning signal is instead sparse and time-consuming to obtain. We evaluate on a collection of 2-D treasure hunting tasks. These tasks require the agent to discover a rule that determines the location of buried treasure in a large collection of environments of the kind shown in Figure 7. To recover the treasure, the agent must navigate (while avoiding water) to its goal location, then perform a DIG action. At this point the episode ends; if the treasure is located in the agent's current position, it receives a reward, otherwise it does not. In every task, the treasure has consistently been buried at a fixed position relative to some landmark (in Figure 7 a heart). Both the offset and the identity of the target landmark are unknown to the agent, and the location of the landmark varies across maps. Indeed, there is nothing about the agent's observations or action space to suggest that landmarks and offsets are even the relevant axes of variation across tasks: only the language reveals this structure.

The interaction between language and learning in these tasks is rather different from the supervised settings. In the supervised case, language serves mostly as a guard against overfitting, and can be generated conditioned on a set of pre-provided concept-learning observations. Here, agents are free to interact with the environment as much as they need, but receive observations only during interaction. Thus our goal here will be to build agents that can _adapt quickly_ to new environments, rather than requiring them to immediately perform well on held-out data.

Why should we expect L\({}^{3}\) to help in this setting? In reinforcement learning, we typically encourage our models to explore by injecting randomness into either the agent's action space or its underlying parameterization. But most random policies exhibit nonsensical behaviors; as a result, it is inefficient both to sample in the space of network weights and to perform policy optimization from a random starting point. Our hope is that when parameters are chosen from within a structured family, a stochastic search in this structured space will only ever consider behaviors corresponding to a reasonable final policy, and in this way discover good behavior faster than ordinary RL.

Here the interpretation model \(f\) describes a policy that chooses actions conditioned on the current environment state and a linguistic parameterization. As the agent initially has no observations at all, we simply design the proposal model to generate unconditional samples from a prior over descriptions. Taking \(x\) to be an agent's current observation of the environment state, we define a state representation network and models:

This parameterization assumes a discrete action space, and assigns to each action a probability proportional to a bilinear function of the encoded description and world state. \(f\) is an instruction following model of a kind well-studied in natural language processing Branavan et al. (2009); the proposal model allows it to generate its own instructions without external direction. To learn, we sample a fixed number of descriptions \(w\) from \(q\). For each description, we sample multiple rollouts of the policy it induces to obtain an estimate of its average reward. Finally, we take the highest-scoring description and fine-tune its induced policy.

At language-learning time, we assume access to both natural language descriptions of these tar

Figure 7: Example treasure hunting task: the agent is placed in a random environment and must collect a reward that has been hidden at a consistent offset with respect to some landmark. At language-learning time only, natural language instructions and expert policies are provided. The agent must both learn primitive navigation skills, like avoiding water, as well as the high-level structure of the reward functions for this domain.

 

get locations provided by human annotators, as well as expert policies for navigating to the location of the treasure. The multitask model we compare to replaces these descriptions with trainable task embeddings.4 The learner is trained from task-specific expert policies using DAgger Ross et al. (2011) during the language-learning phase, and adapts to individual environments using "vanilla" policy gradient Williams (1992) during the concept learning phase.

Footnote 4: In RL, the contribution of L\({}^{3}\) is orthogonal to that of meta-learning—one could use a technique like RL\({}^{2}\) Duan et al. (2016) to generate candidate descriptions more efficiently, or maml Finn et al. (2017) rather than zero-shot reward as the training criterion for the interpretation model.

The environment implementation and linguistic annotations are in this case adapted from a natural language navigation dataset originally introduced by Janner et al. (2017). In our version of the problem (Figure 7), the agent begins each episode in a random position on a randomly-chosen map and must attempt to obtain the treasure. Relational concepts describing target locations are reused between language learning and concept-learning phases, but the environments themselves are distinct. For language learning the agent has access to 250 tasks, and is evaluated on an additional 50.

Averaged learning curves for held-out tasks are shown in Figure 8. As expected, reward for the L\({}^{3}\) model remains low during the initial exploration period, but once a description is chosen the score improves rapidly. Immediately L\({}^{3}\) achieves better reward than the multitask baseline, though it is not perfect; this suggests that the interpretation model is somewhat overfit to the pretraining environments. After fine-tuning even better results are rapidly obtained. Example rollouts are visualized in E. These results show that the model has used the structure provided by language to _learn_ a better representation space for policies--one that facilitates sampling from a distribution over interesting and meaningful behaviors.

## 7 Other Related Work

This is the first approach we are aware of to frame a general learning problem as optimization over a space of natural language strings. However, many closely related ideas have been explored in the literature. String-valued latent variables are widely used in language processing tasks ranging from morphological analysis Dreyer and Eisner (2009) to sentence compression Miao and Blunsom (2016). Natural language annotations have been used in conjunction with training examples to guide the discovery of logical descriptions of concepts Ling et al. (2017); Srivastava et al. (2017), and used as an auxiliary loss for training Frome et al. (2013), analogously to the Meta+Joint baseline in this paper. Structured language-like annotations have been used to improve learning of generalizable structured policies Oh et al. (2017); Andreas et al. (2017); Denil et al. (2017). Finally, natural language instructions available at _concept-learning_ time (rather than language-learning time) have been used to provide side information to reinforcement learners about high-level strategy Branavan et al. (2011), environments Narasimhan et al. (2017) and exploration Harrison et al. (2017).

## 8 Conclusion

We have presented an approach for learning in a space parameterized by natural language. Using simple models for representation and search in this space, we demonstrated that our approach outperforms standard baselines on classification, structured prediction and reinforcement learning tasks. We believe that these results suggest the following general conclusions:

Language encourages compositional generalizationStandard deep learning architectures are good at recognizing new instances of familiar

Figure 8: Treasure hunting reward obtained by each learning algorithm across multiple evaluation environments, after language learning has already taken place (bands show 95% confidence intervals for mean performance). _Multitask_ learns an embedding for each task, while _Scratch_ trains on every task individually. L\({}^{3}\) rapidly discovers high-scoring policies in most environments. Dashed line indicates the end of the concept-learning phase; subsequent performance comes from fine-tuning. The max reward for this task is 3.

 

concepts, but not always at generalizing to new ones. By forcing decisions to pass through a linguistic bottleneck in which the underlying compositional structure of concepts is explicitly expressed, stronger generalization becomes possible.

_Language simplifies structured exploration_. Natural language scaffolding provides dramatic advantages in problems like reinforcement learning that require exploration: models with latent linguistic parameterizations can limit exploration to a class of behaviors that are likely _a priori_ to be goal-directed and interpretable.

And generally, _language can help learning_. In multitask settings, it can even improve learning on tasks for which no language data is available at training or test time. While some of these advantages are also provided by techniques built on top of formal languages, natural language is at once more expressive and easier to obtain than formal supervision. We believe this work hints at broader opportunities for using naturally-occurring language data to improve machine learning for tasks of all kinds.

## Acknowledgments

JA is supported by a Facebook graduate fellowship.

## References

* Andreas et al. (2017) Jacob Andreas, Dan Klein, and Sergey Levine. 2017. Modular multitask reinforcement learning with policy sketches. In _Proceedings of the International Conference on Machine Learning_.
* Bongard (1968) Mikhail Moiseevich Bongard. 1968. The recognition problem. Technical report.
* Branavan et al. (2009) S.R.K. Branavan, Harr Chen, Luke S. Zettlemoyer, and Regina Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_.
* Branavan et al. (2011) S.R.K. Branavan, David Silver, and Regina Barzilay. 2011. Learning to win by reading manuals in a Monte-Carlo framework. In _Proceedings of the Human Language Technology Conference of the Association for Computational Linguistics_.
* Caruana (1998) Rich Caruana. 1998. Multitask learning. In _Learning to learn_, Springer.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. _arXiv preprint arXiv:1409.1259_ .
* Copestake et al. (2016) Ann A Copestake, Guy Emerson, Michael Wayne Goodman, Matic Horvat, Alexander Kuhnle, and Ewa Muszynska. 2016. Resources for building applications with dependency minimal recursion semantics. In _Language Resources and Computation_.
* Denil et al. (2017) Misha Denil, Sergio Gomez Colmenarejo, Serkan Cabi, David Saxton, and Nando de Freitas. 2017. Programmable agents. _arXiv preprint arXiv:1706.06383_ .
* Devlin et al. (2017) Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. RobustFill: Neural program learning under noisy I/O. In _Proceedings of the International Conference on Machine Learning_.
* Donahue et al. (2015) Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional networks for visual recognition and description. In _Proceedings of the Conference on Computer Vision and Pattern Recognition_.
* Dreyer and Esner (2009) Markus Dreyer and Jason Eisner. 2009. Graphical models over multiple strings. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* Duan et al. (2016) Yan Duan, John Schulman, Xi Chen, Peter L Bartlettt, Ilya Sutskever, and Pieter Abbeel. 2016. RL\({}^{2}\): Fast reinforcement learning via slow reinforcement learning. _arXiv preprint arXiv:1611.02779_ .
* Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In _Proceedings of the International Conference on Machine Learning_.
* Frome et al. (2013) Andrea Frome, Greg Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, MarcAurelio Ranzato, and Tomas Mikolov. 2013. Devise: A deep visual-semantic embedding model. In _Advances in Neural Information Processing Systems_.
* Gopnik and Meltzoff (1987) Alison Gopnik and Andrew Meltzoff. 1987. The development of categorization in the second year and its relation to other cognitive and linguistic developments. _Child Development_ .
* Gulwani (2011) Sumit Gulwani. 2011. Automating string processing in spreadsheets using input-output examples. _ACM SIGPLAN Notices_ 46(1).
* Harrison et al. (2017) Brent Harrison, Upol Ehsan, and Mark O Riedl. 2017. Guiding reinforcement learning exploration using natural language. _arXiv preprint arXiv:1707.08616_ .
* Janner et al. (2017) Michael Janner, Karthik Narasimhan, and Regina Barzilay. 2017. Representation learning for grounded spatial reasoning. _Transactions of the Association for Computational Linguistics_ .
* Jia and Liang (2016) Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. _arXiv preprint arXiv:1606.03622_ 

Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. _Proceedings of the Conference on Computer Vision and Pattern Recognition_ .
* Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of the International Conference on Learning Representations_.
* Kitzelmann and Schmid (2006) Emanuel Kitzelmann and Ute Schmid. 2006. Inductive synthesis of functional programs: An explanation based generalization approach. _Journal of Machine Learning Research_ 7.
* Kuhnle and Copestake (2017) Alexander Kuhnle and Ann Copestake. 2017. Shapeworld-a new test methodology for multimodal language understanding. _arXiv preprint arXiv:1704.04517_ .
* Kushman and Barzilay (2013) Nate Kushman and Regina Barzilay. 2013. Using semantic unification to generate regular expressions from natural language. In _Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics_.
* Kwiatkowski et al. (2011) Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2011. Lexical generalization in ccg grammar induction for semantic parsing. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* Lau et al. (2003) Tessa Lau, Steven A Wolfman, Pedro Domingos, and Daniel S Weld. 2003. Programming by demonstration using version space algebra. _Machine Learning_ 53(1).
* Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_.
* Miao and Blunsom (2016) Yishu Miao and Phil Blunsom. 2016. Language as a latent variable: Discrete generative models for sentence compression. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* Narasimhan et al. (2017) Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. 2017. Deep transfer in reinforcement learning by language grounding. _arXiv preprint arXiv:1708.00133_ .
* Oh et al. (2017) Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. 2017. Zero-shot task generalization with multi-task deep reinforcement learning. In _Proceedings of the International Conference on Machine Learning_.
* Raven (1936) John C Raven. 1936. Mental tests used in genetic studies: The performance of related individuals on tests mainly educative and mainly reproductive. _Unpublished masters thesis, University of London_ .
* Ross et al. (2011) Stephane Ross, Geoffrey J Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In _International Conference on Artificial Intelligence and Statistics_.
* Santoro et al. (2016) Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. 2016. Meta-learning with memory-augmented neural networks. In _Proceedings of the International Conference on Machine Learning_.
* Schmidtuber (1987) Jurgen Schmidhuber. 1987. Evolutionary principles in self-referential learning. _On learning how to learn. Diploma thesis, Institut f. Informatik, Tech. Univ. Munich_ .
* Simonyan and Zisserman (2014) K Simonyan and A Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. _arXiv preprint arxiv:1409.1556_ .
* Singh and Gulwani (2012) Rishabh Singh and Sumit Gulwani. 2012. Learning semantic string transformations from examples. In _International Conference on Very Large Databases_.
* Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard S Zemel. 2017. Prototypical networks for few-shot learning. _arXiv preprint arXiv:1703.05175_ .
* Socher et al. (2014) Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. 2014. Grounded compositional semantics for finding and describing images with sentences. _Transactions of the Association for Computational Linguistics_ 2.
* Srivastava et al. (2017) Shashank Srivastava, Igor Labutov, and Tom Mitchell. 2017. Joint concept learning and semantic parsing from natural language explanations. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* Suhr et al. (2017) Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. A corpus of natural language for visual reasoning. In _55th Annual Meeting of the Association for Computational Linguistics, ACL_.
* Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. 2016. Matching networks for one shot learning. In _Advances in Neural Information Processing Systems_.
* Williams (1992) Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine learning_ 8(3-4).

 

Model and Training Details

In all models, RNN encoders and decoders use gated recurrent units (Cho et al., 2014).

Few-shot classificationModels are trained with the adam optimizer (Kingma and Ba, 2015) with a step size of 0.0001 and batch size of 100. The number of pretraining iterations is tuned based on subsequent concept-learning performance on the development set. Neural network hidden states, task parameters, and word vectors are all of size 512. 10 hypotheses are sampled during for each evaluation task in the concept-learning phase.

Programming by demonstrationTraining as in the classification task, but with a step size of 0.001. Hidden states are of size 512, task parameters of size 128 and word vectors of size 32. 100 hypotheses are sampled for concept learning.

Policy searchDagger (Ross et al., 2011) is used for pre-training and vanilla policy gradient (Williams, 1992) for concept learning. Both learning algorithms use adam with a step size of 0.001 and a batch size of 5000 samples. For imitation learning, rollouts are obtained from the expert policy on a schedule with probability \(0.95^{t}\) (for \(t\) the current epoch). For reinforcement learning, a discount of 0.9 is used. Because this dataset contains no development data, pretraining is run until performance on the pretraining tasks reaches a plateau. Hidden states and task embeddings are of size 64. 100 hypotheses are sampled for concept learning, and 1000 episodes (divided evenly among samples) are used to estimate hypothesis quality before fine-tuning.

## Appendix B Dataset Information

ShapeWorldThis is the only fully-synthetic dataset used in our experiments. Each scene features 4 or 5 non-overlapping entities. Descriptions refer to spatial relationships between pairs of entities identified by shape, color, or both. There are 8 colors and 8 shapes. The total vocabulary size is only 30 words, but the dataset contains 2643 distinct captions. Descriptions are on average 12.0 words long.

Regular expressionsAnnotations were collected from Mechanical Turk users. Each user was presented with the same task as the learner in this paper: they observed five strings being transformed, and had to predict how to transform a sixth. Only after they correctly generated the heldout word were they asked for a description of the rule. Workers were additionally presented with hints like "look at the beginning of the word" or "look at the vowels". Descriptions are automatically preprocessed to strip punctuation and ensure that every character literal appears as a single token.

The regular expression data has a vocabulary of 1015 rules and a total of 1986 distinct descriptions. Descriptions are on average 12.3 words in length but as long as 46 words in some cases.

NavigationThe data used was obtained from Janner et al. (2017). We created our own variant of the dataset containing collections of related tasks. Beginning with the "local" tasks in the dataset, we generated alternative goal positions at fixed offsets from landmarks as described in the main section of this paper. Natural-language descriptions were selected for each task collection from the human annotations provided with the dataset. The vocabulary size is 74 and the number of distinct hints 446. The original action space for the environment is also modified slightly: rather than simply reaching the goal cell (achieved with reasonably high frequency by a policy that takes random moves), we require the agent to commit to an individual goal cell and end the episode with a special DIG action.

Data augmentationDue to their comparatively small size, a data augmentation scheme Jia and Liang (2016) is employed for the regular expression and navigation datasets. In particular, wherever a description contains a recognizable entity name (i.e. a character literal or a landmark name), a description template is extracted. These templates are then randomly swapped in at training time on other examples with the same high-level semantics. For example, the description _replace first b with e_ is abstracted to _replace first CHAR1 with CHAR2_, and can subsequently be specialized to, e.g., _replace first c with d_. This templating is easy to implement because we have access to ground-truth structured concept representations at training time. If these were not available it would be straightforward to employ an automatic template induction system Kwiatkowski et al. (2011) instead 

Examples: ShapeWorld

(Examples in this and the following appendices were not cherry-picked.) 

Examples: Regular Expressions

```
Example in: **Example out:** medieval ileideeval leading consonant si replaced with i1
``` Input: **Inser** wafer lifter conventions liberventions hands springsimalsprimes ```
``` uptakes updates bouncing punching embroidery evey owith u n repleace every owith u n regulation rebilliousness stebelliumss stunlight
```

```
fulfifest fluffiest kidapers kid managers acting eating gripping disagreementsdisagreeeents
```

clandestine clandqtime e limming limimg homes hinfelood liffalood where e appears , replace it and the following letter with q ```
``` fruilessly frevitlessly ifthe word ends with an s , replace handler sandier the last two letters with mu numbers newlers wevrelrium dewlaps devalum change last to m uif consomant
```

Ladylike ladylike finitlockml student student surtaxenl surfaces watzenl bedecks hedecknl ```


``` porringgerportinger add e next toletter g package yangage syange curtseyinge whens a letter is preceded by a g .
#onsieurmonsireur enos



Examples: Navigation

White breadcrumbs show the path taken by the agent.

```
Human description: move to the star inferred description: reach the star cell
```

reach square one right of triangle reach cell to the right of the triangle ```
reach cell on left of triangle reach square left of triangle reach square left of triangle reach spade go to the spade
```

Left of the circle go to the cell to the left of the circle reach cell below the circle reach cell below circle