Intra-agent speech permits zero-shot task acquisition
Authors: Chen Yan
Federico Carnevale
Petko Georgiev
Adam Santoro
Aurelia Guy
Alistair Muldal
Chia-Chun Hung
Josh S Abramson
Timothy P Lillicrap
Greg Wayne
Venue: NeurIPS 2022 Accept (2022)

Abstract: Human language learners are exposed to a trickle of informative, context-sensitive language, but a flood of raw sensory data. Through both social language use and internal processes of rehearsal and practice, language learners are able to build high-level, semantic representations that explain their perceptions. Here, we take inspiration from such processes of "inner speech" in humans (Vygotsky, 1934) to better understand the role of intra-agent speech in embodied behavior. First, we formally pose intra-agent speech as a semi-supervised problem and develop two algorithms that enable visually grounded captioning with little labeled language data. We then experimentally compute scaling curves over different amounts of labeled data and compare the data efficiency against a supervised learning baseline. Finally, we incorporate intra-agent speech into an embodied, mobile manipulator agent operating in a 3D virtual world, and show that with as few as 150 additional image captions, intra-agent speech endows the agent with the ability to manipulate and answer questions about a new object without any related task-directed experience (zero-shot). Taken together, our experiments suggest that modelling intra-agent speech is effective in enabling embodied agents to learn new tasks efficiently and without direct interaction experience.

Intra-agent speech permits zero-shot task acquisition
Chen Yan
Federico Carnevale
Petko Georgiev DeepMind London, UK petkoigddeepmind.com com
DeepMind London, UK ywc@deepmind.co com
DeepMind London, UK fedecarrnev@deepiind.cmmm
Adam Santoro DeepMind London, UK adamaantoroodeepmind.com
Aurelia Guy OpenAl San Francisco, USA Taurellaggu0gmaii.cc com
Alistair Muldal DeepMind London, UK alimulda wellddd@e@@emmddee
Chia-Chun Hung
Josh Abramson DeepMind London, UK j jabramson@deepmindd com
Timothy Lillicrap
Isomorphic Labs*
DeepMind London, UK
London, UK al dldenhun@@goggle.comm com
countzero@ddepmind.ccomm
Gregory Wayne DeepMind London, UK gregwrayneddeepmind.ccmm com
Abstract
Human language learners are exposed a trickle of informative, context-sensitive language, but a flood of raw sensory data. Through both social language use and internal processes of rehearsal and practice, language learners are able to build high-level, semantic representations that explain their perceptions. Here, take inspiration from such processes of "inner speech" in humans (Vygotsky) 1934) to better understand the role of intra-agent speech in embodied behaviour. First, formally pose intra-agent speech as a semi-supervised problem and develop two algorithms that enable visually grounded captioning with little labeled language data. We then experimentally compute scaling curves over different amounts of labeled data and compare the data efficiency against supervised learning baseline. Finally, we incorporate intra-agent speech into an embodied, mobile manipulator agent operating in 3D virtual world, and show that with few as 150 additional image captions. intra-agent speech endows the agent with the ability to manipulate and answer questions about a new object without any related task-directed experience (zero-shot). Taken together, our experiments suggest that modelling intra-agent speech effective in enabling embodied agents learn new tasks efficiently and without direct interaction experience.
1 Introduction
Contemporary language models learn from troves of text comprising billions, and sometimes trillions of tokens. This is not the same situation faced by embodied language learners, such human children,
Work done while DeepMind.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Training
Caption about new object (rare)
Self supervision about new object!
Tasks about known objects
a drum
can see
ma.
pear
Inference
Hand the
Figure I: Learning to speak about new objects permits zero-shot motor task behavior. Is it possible for agents perform complex tasks with new object just by learning to name it? In this work, show it is. We first trained agents speak about objects that had never been used any task, named in any interaction, by training agents to caption their observations (top row, left middle). then introduced interactive tasks involving other objects and trained agents with imitation learning (top right). Using only small number of labeled captions that pertained to new object category (drums), agents were able to perform tasks involving the new objects, such as lifting and delivering them second avatar (bottom).
whose raw sensory data vastly exceeds their social-linguistec experiences. Psychologists have long noted the existence of non-social, intra-personal callogue-suchs as inner speech (silent) or private speech (aloud)-whic provides additional linguistic experiences beyond those encountered in social settings. Indeed, one possible reason for perra-peroonal speech may be to support the development of social speech (15)). though the inverse causal relation may also be possible [.
intra-personal speech may also affect broader aspects of our cognition: the process wherein we pri- vately speak about our perceptions subsequently impacts how attend to, remember, manipulate the objects of our perception [7H12]. For example, our short-term memories tend to commit errors for items that phonetically but not visually semantically similar [ [9, and "self-talk" affects task performance on executive tasks both in children [1 and in adults 7 and has been widely used to improve athletic performance (10]
Inspired by these ideas, here we ask if we can leverage intra-agent speech as a tool for artificial intelligence research. Specifically, we ask the following questions: First, is it possible to model intra-agent speech as a machine learning problem, enabling us to make up for a paucity of language data when given access to large amount of unlabeled data? Second, can mechanisms for learning intra-agent speech impact the subsequent behavior of an embodied agent?
Regarding the first, we formally structure intra-agent speech as semi-supervised problem. In particular, we model intra-agent speech samples as variational latent variables representing images, and use this perspective to devise two algorithms enabling visual captioning given a paucity of directly labeled data. Regarding the second, we go on to show how embedding such algorithms in an embodied agent not only permits the capacity to speak about any given perception (that is, caption, which is precisely what is being optimized for) but rather also permits zero-shot behaviors involving new objects that were never directly implicated in any task-directed experience (see Figure T).
2 Approach
At a high level, we structured intra-agent speech as the production of language generated from inputs (which our case manifests as captions of images), with two additional concerns: First, there will be a preponderance of unlabeled data (i.e., pure images) relative to labeled data (i.e.. images with associated captions) in our captioning dataset, much like the situations faced by human children in nature and semi-supervised learning in machine learning. Second, the linguistic representations eventually produced will provide auxiliary supervision for an embodied, acting agent so that they may influence interactive behavior.

---

Intra-agent speech
(BomiSuuanntsnn Pretraning)
-Image
Human Behaviour
Decoder
Provides target for
V Palicy
Encoder trained separately for this work, principle could be
Human Caplion
Caption
D Caption
Human captions avsilable small pertion of
Encoder
Agent
Depicted caption lows. We wio experimented with a contractive "metthhing" loss wherein captions are provided as inputs
Image
Image
Language
Figure 2: Using models of intra-agent speech to influence behavior. During intra intra-agent agent speech pre-traine pre-training. agents reccive image observations as input and produce either utterances that describe the observations (for the small number of inputs for which we have human annotated labels) (using a generative model whose latent representations comprise language tokens). During behavioral training, the intra-agent speech module frozen and used to provide targets an auxiliary captioning loss, whose gradients affect components responsible for action selection.
tasked with reconstructing the images
In humans, the learning of external and internal speech, the rehearsal of inner speech, and the practice of tasks can develop concurrently. In the first stage, we pretrain captioning model by semi-supervised learning, using a small amount of labeled language data and the large amount of agent's 's experience as unlabeled data. This model is able to commentate about visual observations critical function of "inner speech". This model is then used to commentate (via what we have named "intra-agent speech") on further experiences to shape representations as the agent goes on to learn specific tasks, and we show comparisons of the agent's performance with and without this intra-agent speech process.
diagram of the semi-supervised learning approach to intra-agent speech and its role in acquiring behaviors is shown in Figure
2.1 Methods for Semi-supervised Language Learning
Our data D are composed of dataset D, of paired data (x, and dataset Da, of unpaired data {x). In our interpretation, y corresponds to language (a caption) describing image x. When observing unpaired data, treat y as latent variable that must be inferred to explain the observed data [33. We develop two variants, one generative and one contrastive, of the semi-supervised language learning algorithm in what follows.
2.1.1 Generative variant
The preponderance of our data is unlabeled. To model these data we can maximise a variational lower bound on the log marginal distribution, log (x).. corresponding posterior inference. We will use learned approximation to the true posterior distribution, (y x). We call this approximate distribution an Vimage-conditional language encoder" (or sometimes, simply, caption model) since its functional role is to convert images into a discrete latent code that is, ideally, a semantically meaningful language caption:
log (x)) 2
Σ Bynly x) Pog y)] (x x))lps(y)) = Ju- D.
(1)
D
In Equation D the space of latent codes (captions) is discrete and vast, complicating optimization. However, we can form stochastic approximation by sampling from qu (y x) to estimate gradient.
VaJu Σ (yn(yxi)) [Vw log ga (y x) po (x)xxxxy)>>)))) lyg log (y x))]
DD.
For simplicity, we pretrain the prior (y( as language model that learns on of our caption data, and fix it during this optimisation phase. Thus, optimising Ju corresponds an retuopy-regularisee
3

---

RL problem [1. The encoder (y x) serves as a policy that achieves reward for matching the language model prior while enabling the language-condition. decoder reconstruct the image.
While this reward pressures the image-conditional language encoder to produce linguistically mean- ingful latent codes, we can also exert additional pressure by using small number of images with associated captions ("labeled data"), and leverage supervised learning to learn log pa (x y), log (y)),, log qw x): Σ x,y~Dy
J =
[oog po(x yy) + log Ps (y) + log (y x)]
(3)
Over the combined paired and unpaired datasets, our complete objective is to maximise Jp + Ja.
2.1.2 Contrastive variant
The language-conditional image decoder po(x y) is a potentially complicated term model. To understand if this term represents liability for the approach and to generate an algorithm with potentially complementary strengths, we also developed a contrastive, energy- approach, amounting to multi-class classification, approximate it [17119] If we have multi-class classifier that discriminates if a batch element y; is paired to a batch element Xj, and we express cross- entropy loss for the correct index = j as L [( y,, = =)), then we can show that the = f) up to constant generative reconstruction loss log Po xx yj) is proportional to [((x)]__I. 1 b=1 y,, = =) for batch factors with respect to y (Appendix A.1 Thus, we can substitute [
==
element j. log Px(y y,), in Equation and the gradients with respect to w remain the same in expectation.
We represented the softmax in the classifier as ((x)) eloy)
Σ e [(ny] "(xy)" where and g are networks that compress the image into a vector and the caption into a vector, respectively. To train this classifier, adopted similar approach to recent models for visual language representation learning (20 21) by minimizing the sum of two losses, one for matching each image to its paired caption in the batch, and one for matching each caption to its paired image:
B Σ log DB (((x)) L = B
f(x,) R(ys)
B Σ B J=1
é (xx)) R(ys)
+
log
(4)
Σ cf(x,)) gg()))
The networks and g are trained at the same time as the rest of the model, and samples from (y Xz) are included positive examples for the classifier along with the original paired data.
2.1.3 Intra-agent speech architecture and optimization
Image-conditional language encoder. The image-conditional language encoder received input images with resolution 96 x 72. We used the ResNet architecture described in (22). with strides 21, 2) 3 x 3 kernel size, and channel sizes (64, 64, 128, 256) for a total of 20 layers. Language was produced by a 4-layer causal transformer with 256 embedding size and attention heads (2). which attended to the 432 hyper-pixe vectors generated from the ResNet, and produced a sequence 4. 000 token vocabulary. Language targets were encoded with Senten- cePiece byte-pair encoder [24] trained using language data from (25
of logits corresponding
Generative semi-supervised model. The language prior comprised separate transformer pre- trained on all the caption labels in the training data (and hence, did not have image inputs). For the image decoder, first pre-trained VQ-VAE (26) all unlabeled images to define the VQ-VAE's codebook, compressing each image into 432 tokens with a vocabulary of 512. We then used an 8-layer transformer with 512 embedding size with causal masking model the VQ-VAE tokens autoregressively. The conditioning captions were tokenized and embedded as in the image-conditions language encoder, and then processed by distinct transformer with 4 layers, 256 embedding size, and heads, whose output was then cross- attended by the image decoder to produce a reconstruction.
Contrastive semi-supervised model. For the contrastive model, images were first encoded using ResNet with the same architecture as the image-conditional language encoder, and globally mean- pooled into one vector with dimension 1024. Language was first encoded using transformer with

---

the same 4-yyyer architecture described above, and then flattened and compressed with 2-layer MLP with dimensions (2048, 1024).
Optimization. We used V-MPO 16 to optimize (y x). The loss was optimized by the Adam optimizer (27] with B1 0.9, f2 0.999 and learning rate of 2 x 10-1 with early stopping at the lowest log-likelihood over captions in validation data. We trained all models with a batch size of 128 except for the contrastive classifier model, which also received 2, 048 unlabeled images per batch, giving total batch size of 2, 176. We trained our models using Tensor Processing Units (TPUv3) 28 In all experiments the early stopping criteria was reached within 150K optimization steps.
2.2 Agent training
The agent was trained by behavioral cloning large corpus of human interactions in the Playhouse environment [22]. Briefly, it comprises a 3D simulated environment wherein two embodied avatars interact with natural language to cooperatively accomplish tasks involving object manipulation, navigation, and question answering. In addition to receiving visual observations and producing motor actions, agents in this environment also receive language inputs and produce language outputs. This allows them to answer questions, participate in dialogue, ask for clarifications, and so on.
Our approach to training the linguistic representations of the agent therefore has steps. First, by using cemi-supereised language learning, we can amplify the impact of sparse language annotations provided for small number of images in learning the intra-agent speech module. In turn, using this trained module, we are able to sample captions at high frequency as the agent acts, providing the agent with dense language supervision. Therefore, while the agent trains using behavioral cloning, we can sample captions from the pretrained intra-agent speech module, and provide these as either auxiliary inputs targets the agent that they may influence the agent parameters optimized to produce embodied behaviors.
For our first loss the caption loss captions arising intra-agent speech samples were provided as targets for the agent's language output policy:
Lc ΣΣ
Σ In (yryyh,,ooss
(5)
where गy is the language action policy of the agent, y is the caption sample, o represents all visual and text observations, and K represents the maximum unroll length. To allow the agent distinguish whether it was being tasked with emitting a caption of its current observation, or whether it was being tasked with emitting language for the purposes of interactive behavior (see (22)). the input of the language policy is summed with learnable embedding of the indicator variable representing whether the target is captioning or language output from demonstration.
While the caption loss trains the agent to speak about what it sees, the agent's language encoding (via linguistic inputs) remains untrained. To this end we introduced a second loss: the caption-matching loss. In the caption-man mitiion-atccing// loss, the agent is required to predict whether a given caption, provided as input, matches the observed image. Negative samples (i.e., captions that do not match the observation) are captions associated with images from elsewhere in the batch [2)] To perform this prediction, an auxiliary classifier D is attached to the network of the agent that encodes both visual and text observations in separate training pass from the behavioral cloning optimization.
LCM 1 ΣΣΣ In D(0,,, vo.)) + In (1 D ())n()))))
Here x represents the visual observation, and the caption sample is fed text observation. The roll() function rolls the index over batches, so that Faal((s represents a caption from another batch
element as negative example. The roll function
implemented as roll(b) (b + 1) mod B.
And finally, the behavioral cloning loss over human language and movement action sequences is as described in Interactive Agents Team 22
LBC
In m (a), + In Tm (a)) ob.
5

---

where the Tm and a" represents the movement policy and action, respectively.
The total loss is the sum of these losses I = Lo+ LCM + Luc. Other than the addition of the caption loss and caption-matching loss, we follow Interactive Agents Team 22 for the agent architecture and training hypper-parameter,,
Experiments
3.1 Semi-supervised captioning
Hypothesis. We first sought to validate our method of semi-supervised learning. We hypothesized that with the additional unlabeled images, the semi-supervised model would be more data efficient (with respect to labelled data) than supervised baseline.
Data. As mentioned, the domain in which tested our methods is called the Playhouse, which originated in Interactive Agents Team [22]. The authors compiled approximately three years worth of interaction data, comprising about billion image frames. The authors have confirmed no personally identifiable information or offensive content is contained in the dataset and gave consent for us to access As there are no associated captions with any frames, we treat each frame in this dataset as a single image, and used all the data as our "unpaired" dataset.
For the "paired" dataset, we engaged with crowd raters to provide corresponding captions for 78K uniformly sampled images from the unpaired dataset. Raters were instructed to describe what they saw in the image, with particular reference randomly selected object indicated in bounding box. Detailed instructions are included in Appendix [A.2] On average, each image contain 12.9 objects, and its corresponding caption describe 1.2 of them.
Evaluation. We measured the log probability of captions in the validation set, CIDEr score 30), object precision and "color-object pair precision". As the images were generated from the Playhouse environment, we had access to the ground truth object identities and colors present in each image, allowing us to check if any object and its color mentioned in caption were indeed present in the image. We calculated object precision to be the proportion of correct objects among all mentioned in sampled captions, normalized by the same calculation from human captions. The color-object pair precision is calculated in the same manner, except for any mentions of an object and its color. As there usually many objects in an image and captions tend to mention just a few, the recall counterpart of this metric (number of correct color-object pairs divided by those present in the image) is not very informative. Captions were greedily sampled for the calculation of CIDEr score object precision and color-object pair precision.
Results. Figure 3 shows sample data used to train the models, well as samples generated from the generative semi-supervised model. Models were able to both produce coherent and relevant captions given an image (e.g., "I can see white bed where a green ball and a green duck are on it") and also generated realistic visual depictions of the Playhouse when conditioned on a language description.
Both semi superviie methods performed better than the supervised baseline across log prob, CIDEr and color-object pair precision, including settings where we artificially constrained the size of the labeled dataset (Figure 4 Notably, when using the full amount of data our semi-supervised methods exceeded human color-object pair precision. To further put the performance gain in context, by leveraging the unpaired dataset of images the semi-supervised methods performed at level equivalent to training supervised model on 3x more labeled data (31.
Interestingly, the semi-supervised methods perform similarly in the object precision metric, despite an advantage in color-object pair precision. This result suggest that the semi-supervise methods may have learned to bind color objects from the unlabelled images, and this results performance gain compared to pure supervised methods.
3.2 Learning captions of new objects
Hypothesis. Upon validating our methodology and amassing the relevant data, we were well-poised to ask the following questions: do semi-supervised methods of intra-agent speech allow model to quickly learn to speak about new object with little supervision?

---

white Unlabeled data
Caption samples
Language conditioned image samples
there while
potled plant in the
green ball and grean
liiing the
Labeled data
there is pink color plane which placed In front of the pink color rack
can sie a olive green oolor bus on the froe in front of tlue color chair and near the violet color dudk
can see a vicket oolor tatile and whill bell on on the left there is grean bed, soddy on floor.
Data and Model Samples. Depicted are data used to train the semi-supervised model (left) and samples from the caption model (middle) and generative image model (right). As seen from the captions, generative modeling of images using a pre-trained language prior, in conjunction with auxiliary supervised training on small number of labeled samples, is sufficient to produce meaningful and relevant language captions of images. Moreover, the model is able to generate realistic-looking scenes from provided captions. See Appendix A.3 for more samples.
Figure 3:
Logp
CIDEr
Object praciaton pharer rommille))
Color-object pair precision Jrarer nemulaad)
0.50
-30
0.55
0.8
-30
0.50
0.5
-10
0.45
0.4
0.4
17.5 0.40
sngeeaddd brnguage
Superviaed
Figure 4: Scaling performance of semi-supervised models over labeled dataset sizes. Semi-supervised methods are trained with unlabeled data and titered amounts of labeled image-caption pairs. Here we calculate the log probability of validation set captions, the CIDEr score (30), object precision and color-object pair precision. Ohject precision (human normalized) calculated by dividing the probability of any mentioned object the image with that of the human ground truth. Color object pair precision (human normalized) is calculated similarly, except for any mentions of an object with its color. Lines represent means from 3 random seeds.
Data. To address the hypothesis, we first chose an object - a drum - and filtered our labeled caption data to remove all instances where the drum is either depicted visually or spoken about verbally. While not strictly necessary, from the unlabeled dataset we also removed all instances where the underlying (and unobserved) instruction included mention of drum, which ensured that all instances of drum appearances were truly "passive"" For example, the unlabeled data may have included trajectories of experience wherein players was tasked with removing ball from shelf, but there might have also happened to drum in view in the room.
Evaluation. We evaluated the model using three different metrics. We first created validation dataset of all captions mentioning drum with their associated image, and evaluated the model's average log likelihood on this drum dataset (drum caption log likelihood). The drum recall, which the probability that them model mentions drum given that a drum is present, measures how the model recognizes drums in the images. We also calculate its counterpart, the probability that a drum present given drum is mentioned, as the drum precision, to measure the accuracy of any drum mentions from the model. The perfect scores for drum recall and drum precision are as result of titering the drum data: we have removed all the data where drum is in the image or caption, and added back small amount where drum is mentioned in the caption. This has consequence that for any image with drum (although very few), its caption will always mention drum
Results. We then performed the same experiment as in Section 3.1 at varying levels of re-introduced labeled drum data, as seen in Figure 4 As expected, the model's ' ability to speak about drums scales with the amount of labeled drum data with which it is trained. Notably, the model's 's recall reaches approximately 70% given only 585 labeled samples of drums. In contrast, the supervised method, where the new drum data is trained in concatenation with the non-drum data, reaches just over 20%. The increase of drum recall in the semi-supervised method is also accompanied with an increase of precision, where the semi-supervised method reaches 80%, versus 40% of the supervised method. These results indicate that the supi-super methods are able to learn about new objects, implicitly, by virtuc of their language-conditionn reconstruction objective. That is, the mere presence of drums
7

---

in the visual observations was sufficient, in conjunction with small amount of labeled drum captions, elicit linguistic understanding of drums as relevant objects in the Playhouse.
Drum caption log likellhood logPidrum coptions)
Drum recall
Drum precision
P/crum mentianedjorum present)
P/crum presentjerum mentioned)
1.0
1.0
-40
0.6
0.6
-45
0.4
-50
0.2
0.0
0.0
72
144
280
576
72
144
576
576
Drum data size
Generative semt seml-supervised
Supervised
Figure 5: Scaling performance of our semi-supervised methods over drum dataset sizes. We examined how the semi-supervised and supervised methods scaled across different amounts of labeled data for novel object. The semi-supervised trained model had a higher log likelihood, was likely to recognize a drum in the image (high true positive rate) while displaying a comparable error-rate (same false positive rate). Extrapolating from supervised true positive rate, the supervised method would need more than 300 more data to reach the same performance as the semi semi-supervised model. Lines represent means from 3 random seeds.
3.3 Learning to manipulate a new object
Hypothesis. As final test tackled the final question: Can the ability quickly learn to speak about an object confer advantages for learning behaviors involving the new object (such as manipulating the object, or answering questions about it)?
Methods and data. To answer this question we leveraged the semi-supervised models built in Section 3.2 to provide auxiliary intra-agent speech objectives. While we employed a two-stage process (learning to caption, and subsequently learning from those captions to shape behavior) for expedience and ease of experimental setup, in principle the two learning procedures could occur in parallel in a unified agent. The agent then proceeded with behavioral cloning- based training as in Interactive Agents Team [22] on data that was filtered not include any drum-based instructions (resulting in 9, 344 removed interactions). While after filter the dataset do not contain any demonstration of drum interaction, in some episodes the drum object is visible (larger than a pixel) in the background in 4.4% of the image frames.
Evaluation. We evaluated agent performance two tasks: in the "lift drum" task, the agent receives the instruction "lift a drum", and required to find the drum in a randomized Playhouse environment and lift it. In the "ask color drum" task, single drum is spawned in the environment. The agent receives the instruction "what is the color of the drum?", and is then required to output language that answers the question (See Appendix A.44 for details). At the end of each episode, the task environment gives out a score of 0 or 1, indicating whether the task is completed successfully (note that these scores are used for evaluation purposes only, and not for training). We collected human performance on each of the tasks. Our participants receive an average of 0.98 for the "lift a drum" task, and 0.51 for the "ask color drum" task. The low score of "ask color drum" is primary due to the stringent reward function that require exact match of the canonical color name, and the fact that many colors similar the canonical name (for example, the canonical "cyan" color is can often referred as "light blue" by human participants). We calculate the final score by dividing the agent score by the average human score, so that final score of represents no success, and a represents human-level performance.
Results.
As seen in Figure 6 a baseline agent trained without any based instructions performs poorly on drum-based evaluation tasks, indicating that there is minimal transfer to be had when learning from the base interaction dataset. On the other hand, a model trained with drum-based instructions (that is, one that trains on the 9, 344 episodes that we had filtered) displays an expected upper bound for this agent of just over 75% on "lifting drums", and just under 80% for answering questions about a drum's 's color, similar to previously reported results (2
8

---

Agents trained with both the caption loss and caption- matching loss, but without any drum-based interaction data achieve approximately 70% of the performance attained by models trained directly on drum-based interaction data. To confirm the generalisability of our methods, we repeated our experiments on the teddy bear object, and observed similar result (Appendix AA.5).
This is a substantial increase over the baseline model, and directly indicates that learning to speak about an object, which is afforded by training model on as few as 585 labeled examples using semi- supervised methods, allows an agent to subsequently exhibit behaviors that implicate the manipulation or mention of the object, without affecting performance on other known objects (Appendix A..))
We then tested whether we could reduce the number of labeled captions even further, and observed that with as few 150 labeled captions we observed similar performance drum-color question answering tasks, and only a slight decrease in drum lifting task performance.
Lift drum
Ask color drum
Lift drum
Ask color drum
0.75
0.75
0.75
0.75
Procedural score (huran nomalzzod maizaazdd
Procedural (human nomalzo
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
Seas
G150
G5s5
Sees
G15g
Gsss
baseline without drum interaction, +C caption loss, caption matching loss,
+D drum interaction data, Seas supervised caption model with 585 annotations drum,
G1100,005 generative semi-supervised caption model with (150, 585) annotations drum
evaluated agent performance on two tasks that require interaction with the novel object (drum): in the first, the agent was tasked to locate and lift drum, and in the second, the agent is required to mention the color of the drum using language. Our model with both the caption loss and caption matching loss (+CM) reaches reward, comparable to approximately 70% of the performance of an agent trained with an additional 9344 episodes of interaction data including drums (+D; left two panels). This result also suggests the crucial role of semi- semi-supervise supervised training in zero-shot gencralization (right two panels); with just 150 labeled captions of drum (G150), our model only displays slight decrease in performance, but still significantly outperforms the supervised model trained on all the labelled captions ($585).
Figure 6: Zero-shot task-directed interaction with a novel object.
than 0.5 human normalized
While we created two procedural tasks to quantitatively evaluate the agent, due to the diversity of the interaction data we used to train the agent, we hypothesized that the agent can perform more tasks with the novel object. Figure summarizess a typical interactive session with our main agent ("+CM" from Figure ( where the agent is asked manipulate the novel object drum, and answer questions about it. The agent can recognize and manipulate the novel object under variety of tasks. This further confirms that our method is general to many background tasks that the agent is able to perform.
Whatks dw color doint
the cuam on bal
Handira diTs
Whare the chan FEX?
Iw dóge
Taka the dhan to da bechoot.
Pu duumt reot gaiass
Plu galan on De Shars
Figure 7: Sample frames from human-agent interaction. With our caption matching loss and caption loss, the agent trained without any drum-based instructions is able manipulate the novel object drum and answer related questions under variety of tasks.

---

4 Related Work
Components. The modeling in this study builds on components developed for generative modeling (VQVAE-2: (2] and contrastive learning (CLIP: (20).
Task generalization in embodied agents. Our work is in the tradition of grounded language research virtual environments [33/35]. Recently, there have been efforts to demonstrate zero- shot task acquisition using variety of means including with the use of pretrained models for language encoders [36] or visual-language models for zero-shot navigation [37]. The majority of
with some recent studies exploring generalization to new tasks given background knowledge from previous tasks [22] 40). Similar phenomena of zero-shot generalization in supervised classification and object detection [20 [1 [1], these studies have often relied on a large, diverse dataset providing supervision where held-out tasks represent novel recombination of parts of the training dataset (e.g.. "compositional" generalization or systematicity 422 Another line of research has focused on few- shot generalization 43 46 through meta-learning over distribution of tasks with generalization to other tasks within the distribution's 's support. Our work is complementary to both lines of work and presents means by which different learning objective can embody "mental practice" enabling generalisation.
these works have focused on generalization to novel phrasings of task instructions 38 (3]
Semi-supervised learning with natural language. The semi semi-superrvied methods we used to train the intra-agent speech module are related to many previous methods that use language as a latent variable. For example, Kingma et al. [13] developed semi-supervised, variational autoencoder ( (VAE) with discrete latent variables, and this work has also been extended to use language as the latent variable. VAES with language latent variables have also been studied in language summarization [47), and in the context of multi-agent communication with natural language 48 49). Our work may present a valuable contribution to this area, showcasing the utility of modern generative and contrastive models in supporting data-efficient learning. Our work's analysis of the scaling curve [31] properties of semi-supervised captioning 520 may provide additional analytical tools for the field.
Lunguage-conditionee image generation. The use of language to create sophisticated, controllable generative models has rapid development in recent years. Recent works based on diffusion models 553 and autoregressive models have produced some of the most impressive results. Notable work includes the DALL-E family of models [56] [57 and GLIDE 58) Other work in the area has also employed VAES 59 and GANs 60)644
Conclusions and Future Work
In this work we built model for intra-agent speech by leveraging semi-superviee learning techniques. We have first shown that the cum-suueee methods outperforms the supervised model in captioning by leveraging unlabeled images. We further demonstrated that with 150 captions of novel object, such models can help embodied agents execute interactive tasks with the novel object without having to learn from direct interactive experience (zero-shot), reaching 70% performance of a baseline agent trained with 9, 000 interaction episodes.
Although we developed the method in a virtual environment, it can be easily generalized to real world applications, applied with either good or bad intent. Embodied language agents acting as robots could be instructed to perform a wide manner of tasks, including harmful ones. Thus, if this work is to be applied in real settings, we advise extra caution minimize unintended or harmful consequences.
This work is a first step into leveraging ideas from the psychology of inner and private speech in humans. While our source of speech was captioning, humans have a richer repertoire of techniques for intra-personal dialogue that affect modalities beyond vision. Direct next steps are, therefore, to explore possible role of intra-agent speech in abstracting over time, and observing its effects on planning and reasoning.
Acknowledgments and Disclosure of Funding
The authors would like to thank Felix Hill for critical discussion, Nathaniel Wong for help with the development of scripted probe tasks, Chris Dyer and Daan Wierstra for reviewing and comments on the manuscript, and Alex Goldin and Guy Scully for organizational support.
10

---

References
[1] Lev S Vygotsky. Thought and language. MIT press, 1934.
[2] Adam Ed Winsler, Charles Ed Fernyhough, and Ignacio Ed Montero. Private speech, executive functioning, and the development of verbal self-regulation. Cambridge University Press, 2009.
[3] Patti Adank, Peter Hagoort, and Harold Bekkering. Imitation improves language comprehension. Psychological Science, 21(12):1903-1909 2010.
[4] Conchi San Martín Martínez, Humbert Boada Calbet, and Peter Feigenbaum. Private and inner speech and the regulation of social speech communication. Cognitive Development, 26(3): 214-229, 2011.
[5] Alan D Baddeley, Susan E Gathercole, and Costanza Papagno. The phonological loop as a language learning device. Exploring Working Memory, pages 164-198, 2017.
[6] Adam Winsler, Rafael Diaz, and Ignacio Montero. The role of private speech in the transition from collaborative independent task performance in young children. Early Childhood Research Quarterly, 12(1):59-79, 1997.
[7] Ben Alderson-Day and Charles Fernyhough. Inmer speech: development, cognitive functions, phenomenology, and neurobiology. Psychological bulletin, 141(5):931, 2015.
[8] Alan D Baddeley. Short-term memory for word sequences as a function of acoustic, semantic and formal similarity. Quarterly journal of experimental psychology, 18(4):362-365, 1966.
[9] Diane Schiano and Michael Watkins. Speech-like coding of pictures in short-term memory. Memory & Cognition, 9(): 10-114, 1981.
[10] Antonis Hatzigeorgiadis, Nikos Zourbanos, Evangelos Galanis, and Yiannis Theodorakis. Self- talk and sports performance: meta-analysis. Perspectives on Psychological Science, 6(4):
348-356, 2011.
[11] Virginia S Tinsley and Harriet Salatas Waters. The development of verbal control over motor behavior: A replication and extension of luria's 's findings. Child Development, pages 746-753, 1982.
[12] Charles Fernyhough and Emma Fradley. Private speech on an executive task: Relations with task difficulty and task performance. Cognitive development, 20( (0(1):103-120 2005.
[13] Durk Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, semi-supervised learning with deep generative models. Advances in neural information processing systems, 27, 2014.
[14] John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-leaming. arXiv preprint arXiv: 2017.
[15] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:/805.00999 2018.
[16] H Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, et al. V-mpo: On-policy maximum posteriori policy optimization for discrete and continuous control. arXiv preprint 1909:: 12238, 2019.
[17] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pages arXiv-1807, 2018
[18] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in Neural Information Processing Systems, 29, 2016.
[19] Chris Dyer. Notes on noise contrastive estimation and negative sampling. arXiv preprint arXiv.1410.8251, 2014.

---

[20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 1748-8763. PMIR, 2021.
[21] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- language representation learning with noisy text supervision. International Conference on Machine Learning, pages 4904-4916. PMLR, 2021.
[22] DeepMind Interactive Agents Team. Creating multimodal interactive agents with imitation and 2021.
self-supervised learning. arXiv preprint
[23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[24] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprinl urXiv: 0626666266 2018.
[25] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.1144 1446, 2021.
[26] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[27] Zijun Zhang. Improved adam optimizer for deep neural networks. In 2018 JEEE/ACM 26th International Symposium on Quality of Service (IWQoS), pages 1-2. IEEE, 2018
[28] Google Cloud. Cloud tensor processing units (tpus), 2022. Itttps://clod. google..ooog...oooo com/
tpu/ /tc///ccs/t/tt//// tpus [Accessed: 2022-05-19].
[29] Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovié, Jason Ramapu- ram, Jeftrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. Advances in Neural Information Processing Systems, 33:25-37, 2020.
[30] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[31] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar arXiv:2001.08361, 2020.
[32] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelin images with vq-vae-2. Advances neural information processing systems, 32, 2019.
[33] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang. Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg. Denis Teplyashin, et al. Grounded language learning in a simulated 3d world. arXiv preprint vrXiv:1706.0655)) 2017.
[34] Maxime Beevalier-Boisverr Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprini 18Xiv:1800.082 2018.
[35] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132. PMLR, 2020.
[36] Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instractiio with deep reinforcement learning via transfer-leaming from text. arXiv preprint arXiv:2005.080 099322 2020.
12

---

[37] Ziad Al-Halah, Santhosh K. Ramakrishnan, and Kristen Grauman. Zero experience re- quired: Plug & play modular transfer learning for semantic visual navigation. arXiv preprint 0rX1v:2202.02440. 2022.
[38] Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation, Conference on Robot Learning, pages 1303-1315 PMLR. 2022.
[39] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint orXiv:2005.07648 2020.
[40] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002. PMLR, 2022.
[41] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoc V Le. Combined scaling for zero-shot transfer learning, arXiv preprini arXiv:2111 11. 10050, 2021.
[42] Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. A benchmark for systematic generalization in grounded language understanding. Advances in neural information processing systems, 11119986-19877 2020.
[43] Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-lcarning. In Conference on robot learning, pages 357-368. PMLR, 2017.
[44] Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Picter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017.
[45] Allan Zhou, Eric Jang, Daniel Kappler, Alex Herzog, Mohi Khansari, Paul Wohlhart, Yunfei Bai, Mrinal Kalakrishnan, Sergey Levine, and Chelsea Finn. Watch, try, learn: Meta-learning from demonstrations and reward. arXiv preprint 190.:33906. 2019.
[46] Kourosh Hakhamaneshi, Ruihan Zhao, Albert Zhan, Pieter Abbeel, and Michael Laskin. Hi- erarchical few-shot imitation with skill transition models. arXiv preprint vrXiv:2107.0898) 2021.
[47] Yishu Miao and Phil Blunsom. Language as latent variable: Discrete generative models for sentence compression. arXiv preprint arXiv: 07, 2016.
[48] Jason Lee, Kyunghyun Cho, and Douwe Kiela. Countering language drift via visual grounding. arXiv preprint arXiv:I909.000
2019.
[49] Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. arXiv preprint vrXiv:2055.00064 2020.
[50] Wenhu Chen, Aurelien Lucchi, and Thomas Hofmann. semi-supervised framework for image captioning. arXiv preprint
2016.
[51] Bicheng Xu, Weirui Kong, and Jiaxuan Chen. Semi-supervised image captioning via recon- struction. In Proceedings of the Intern. Conf. on Comp. Vis.(ICCV), pages 4135-4144, 2017.
[52] Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, and In So Kweon. Image captioning with very scarce supervised data: Adversarial semi-super learning approach. arXiv preprint 1rXiv: 002.//999.0220,1.. 2019.
[53] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[54] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pages 2256-2265 PMLR, 2015.
13

---

[55] Yang Song and Stefano Ermon. Improved techniques for training score- based generative models. Advances neural information processing systems, 33:12438- 12448, 2020.
[56] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831 PMLR, 2021.
[57] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[58] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Πya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112. 2021.
[59] Tiago Ramalho, Tomák Kočiský, Frederic Besse, SM Eslami, Gábor Melis, Fabio Viola, Phil Blunsom, and Karl Moritz Hermann. Encoding spatial relations from natural language. arXiv preprint arXiv: 1 1807,01670 2018.
[60] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text image synthesis. In International conference machine learning, pages 1060-1069. PMLR, 2016.
[61] Tao Xu, Pengehuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 1316-1324, 2018.
[62] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5802-5810, 2019.
[63] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 333-842, 2021.
[64] Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderraman, and Shihao Ji. Improving text-to-image synthesis using contrastive learning. arXiv preprint arXiv:2107.02423, 2021.
14

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ?.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] The code and
the data are proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi-
ments multiple times)? [Yes] See Figure 4 and 6
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3.1 We trained models using Tensor Processing Units (TPUv3), in all experiments the early stopping criteria reached within 150K optimization steps.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We have cited Interactive Agents Team [22].
(b) Did you mention the license of the assets? [N/A] The assets proprietary.
(c) Did you include any new assets either in the supplemental material as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [Yes] Consent for using the data was obtained from Interactive Agents Team [22 under proprietary contract. See Section 3.1
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section 3.1 Interactive Agents Team 222 has confirmed no personal identifiable information or harmful content contained in the dataset.
5. If you used crowdsourcing conducted research with human subjects...
15

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See Appendix AA2
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] The full details of our study design were reviewed by our institution's independent ethical review committee. See Appendix [A.2 (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [No] The compensation rates were reviewed by our institution's ' independent ethical review committee as part of the study design. See Appendix
A.2 A.2
16


Reviews:
[{"review_id": "t1PGjZeGu6_", "reviewer": "Reviewer_2NEi", "summary": "This paper is a novel proof-of-concept of embodied agents gaining the ability of new tasks with two kinds of semi-supervised learning mechanisms, namely 'intra-agent speech'. The results are three-fold - embodied interaction of agents within a simulated environment (or intra-agent speech of unified agent): (1) enhances the data efficiency of the image captioning subsystem of the agent, (2) reduces the number of data required to learn about the new object, (3) allows motor task involving the new object.", "questions": "- Where can I find the image captioning performance of the model trained with only semi-supervised pretraining (i.e., intra-agent speech without an agent)?\n- What does the 'indicator embedding' in line 151 refer to?", "limitations": "The reviewer is unsure whether this work shows that the inner speech of the agents is helpful in efficiently learning about the world as humans do. Unless there is a sufficient amount of experiments and literature in a similar field, such over-claiming should be avoided.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This work largely unifies the frameworks from image captioning, image generation, and reinforcement learning to show the newly devised concept 'intra-agent speech' can work as an effective semi-supervised learning mechanism.\n- The writing is clear enough for readers to understand what hypotheses are established and how they are tested.\n\nWeaknesses\n- The conclusions derived from the experiments aren't backed with enough numerical details. For instance, how should the readers conclude that the proposed methods allow a model to 'quickly' learn with 'little' supervision? To meet a broader audience, those ambiguities should be resolved.\n- The zero-shot skill acquisition task (section 3.3) is only experimented with a single specific object (drum), which is not convincing enough.\n- 95% confidence intervals are obtained with only 3 experiments, which does not seem to be enough sample size. The reviewer considers 5 as a bare minimum for such statistics. (It's also mistyped - \"confident interval\" should be \"confidence interval\")", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mJU_OzXdr7P", "reviewer": "Reviewer_3uFs", "summary": "The authors present an \"intra-agent speech\" framework combining a semi-supervised image captioning model (herein M1) and an agent grounded in the Playhouse environment which takes action and answers questions (herein M2). M1 is trained first on a small amount of labeled data in the Playhouse domain, and its outputs on novel Playhouse samples are used to provide supervision to M2 -- an implementation of the \"intra-agent speech\" idea. The authors present results on 1) M1's semi-supervised captioning data efficiency, 2) M1's low-shot captioning data efficiency for novel objects, and 3) M2's low-shot performance for novel objects. They argue that this implementation of intra-agent speech provides data-efficient supervision to M2 for both language use and acting in the world.", "questions": "1. **Model design:** What is the motivation for your intra-agent speech implementation? A more obvious setup (to me) would be to have the outputs of a captioning model directly affect the representation of the agent, rather than serving as supervision in an auxiliary task. This would seem to resemble more closely the \"intra-agent speech\" idea -- in that the inner language is part of / directly influences mental representations online. While I'm not so attached to this alternate implementation, it is important for me to know why you chose your particular implementation. Can you explain why you chose this auxiliary task route? Why might it be the most effective and/or cognitively faithful method for implementing intra-agent speech?\n\t1. For what it's worth, I'd strongly prefer a thorough discussion on this design choice and alternatives in place of the ~1.5 pages dedicated to the generative vs. contrastive training setup.\n2. **3.1 evaluation:** Why is \"color-object accuracy\" the right custom metric to develop here? I have many other more stringent questions I'd want to ask about a captioning model in this domain. For example:\n\t1. Of the concrete nouns mentioned in a caption, how many have referents in the scene?\n\t2. For noun phrases \"the X,\" how often is there a unique referent selected by X in the scene?\n3. **3.2 training:** What is the training method here? Is the model being fine-tuned with drum examples, or retrained afresh with a dataset that is the union of the original dataset with some fixed number of drum examples?\n\t1. Possibly related question -- what happens to the log-probability of captions for non-drum images as the drum-training proceeds?\n4. **3.2 evaluation:**\n\t1. What is the ideal \"drum true positive rate?\" Captions needn't include every object in a scene. / Humans won't always mention a drum when it appears in an image. Why is 70% better than 20%?\n\t2. The drum false positive rate result doesn't look so interesting to me. There may not be any significant differences between semi-supervised and supervised training. I'd rather see here the other conditional P(drum present | drum mentioned).\n5. **3.3 evaluation:**\n\t1. Over what trajectories are human reward labels collected? Does it include ground-truth captions and ground-truth motions? This seems very important for making sense of the absolute value of the rewards shown in figure 6.\n\t2. Can you show the full distribution of human reward labels, without normalization? (At least in an appendix.)\n\t3. How does the \"ask color\" task work? This doesn't seem analogous to the captioning task imposed on the agent during training.\n6. **Arbitrary evaluation choices:**\n\t1. Why drums? Do you plan to repeat the analyses for other objects? You could have a leave-one-out style evaluation for all objects that appear (or appear with low frequency, at least) in the unlabeled data.\n\t2. Why just lifting? How many motion tasks are there available to test?\n\t\t1. What about a task that is more semantically complicated (e.g. \"put the drum on the table\") or motorically complicated (e.g. \"balance the drum on its side\")?\n7. **Interpretation:** Why does this work? I think the paper would be much more compelling if the authors could provide detail (through a model comparison, representational analyses, or error analysis, for example) on what the auxiliary captioning objective (AKA intra-agent speech) is doing for the model. The particularly interesting result is that the captioning objective affects not just the linguistic output of the agent but also its motion behavior.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper presents an interesting idea motivated by a psychological concept, but the execution and presentation are far from thorough. The model design and evaluation seem to contain several arbitrary choices, and little analysis in the paper helps the reader understand *why* the model improves concretely.\n\nI do not follow this subfield closely and can't speak to the originality or significance of this paper with confidence. I do regard it as significant in that it makes a good-faith attempt to implement a psychologically motivated concept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_HSNCxDX9QA", "reviewer": "Reviewer_Dfhj", "summary": "This paper proposes using image captioning (i.e. intra-agent speech) as an auxiliary task to help embodied agent to execute tasks involving novel objects which are not in the training interaction dataset. In order to provide captions as the target for training agent’s policy, this paper proposes two semi-supervised methods to learn captioning models. The experiments show that 1) the semi-supervised methods can learn to caption with fewer data compared to supervised learning and 2) learn to caption a new object from hundreds of examples to shape the policy with the learned captions to interact with that novel object.", "questions": "- In the experiment of section 3.3, the authors only use the drum data to train the caption model and then use this caption model to train the agent without drum interaction data. In the second training stage, does the agent see and caption the drum? If not, it is unlikely the agent can learn to interact with this new object. How much interaction data in this experiment contains a drum as a background object? This would be a key contributor to the zero-shot performance.\n- Do the captions cover most of the visible objects in the scene? The proposed approach will perform better if the caption has higher coverage of objects. How robust is the proposed model when having lower object coverage?", "limitations": "- As the authors suggest, the proposed intra-agent speech can only generalize to object identification tasks as the captions and observations are visual and object centric.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Novelty: The idea to use captioning as a representation to improve policy generalization is interesting.\n- Data efficiency: The proposed semi-supervised method can caption images with better performance with fewer annotations compared to supervised learning.\n- Explainability: The auxiliary captioning task can potentially help explain the agent’s perception and relates the linguistic representation with the generated policy. However, the paper doesn’t show how the generated captions relate to the success or failure of the agent’s policy.\n\nWeaknesses\n- For the proposed agent to work, it requires specific setups which may be easy to break.\n  1) the caption model needs to be relevant to the tasks the agent will perform. In this paper, they are about identifying objects and their colors. So, the object and color information must be included in the caption annotations the example annotation shows. It would be helpful to see how the agent’s performance relates to how captions match the agent’s task. \n  2) the target object needs to be in the background (see Questions), it is unclear how many times the agent needs to see the object in the background in order to interact with it", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZCMq2BpwocO", "reviewer": "Reviewer_G4n2", "summary": "In this paper, the authors model intra-agent speech as latent codes for embodied agents. In optimizing the ELBO, the authors use a frozen language model to enforce a nautal language like latent code distribution. They test this model on image caption tasks with both paired data and unpaired data, and find with the same amount of unpaired data, semi-supervised could benefit from unpaired data. They also find that the model can learn to speak about objects that are rare in the paired data and only passively observed in the unpaired data. Finally, they find that just through adding “drum” related captioning data, the agent can learn to lift drum. \n", "questions": "Most of the zero-shot results are only on a single concept -- drum. What are the obstacles preventing the authors from conducting the experiments on other concepts as well? ", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe intra-agent speech concept has rarely been discussed in the embodied agent literature. \nTheir finding about generalizing to zero-shot interaction tasks from captioning data is very interesting. \n\nWeakness:\n\nUsing variational methods for semi-supervised learning is not a new idea. The first finding on the benefit of unpaired data is expected. This may not necessarily be a weakness, but branding it as “intra-agent speech” might be misleading and hyping. Since there’s no evidence that this is similar to the human inner speech, it should be made clear that this work has nothing to do with inner speech, and is just a latent variable model. My scores are based on the assumption that the authors could make the amendment to make this clear. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept