Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes using image captioning (i.e. intra-agent speech) as an auxiliary task to help embodied agent to execute tasks involving novel objects which are not in the training interaction dataset. In order to provide captions as the target for training agent’s policy, this paper proposes two semi-supervised methods to learn captioning models. The experiments show that 1) the semi-supervised methods can learn to caption with fewer data compared to supervised learning and 2) learn to caption a new object from hundreds of examples to shape the policy with the learned captions to interact with that novel object.
strengths_and_weaknesses: Strengths
- Novelty: The idea to use captioning as a representation to improve policy generalization is interesting.
- Data efficiency: The proposed semi-supervised method can caption images with better performance with fewer annotations compared to supervised learning.
- Explainability: The auxiliary captioning task can potentially help explain the agent’s perception and relates the linguistic representation with the generated policy. However, the paper doesn’t show how the generated captions relate to the success or failure of the agent’s policy.

Weaknesses
- For the proposed agent to work, it requires specific setups which may be easy to break.
  1) the caption model needs to be relevant to the tasks the agent will perform. In this paper, they are about identifying objects and their colors. So, the object and color information must be included in the caption annotations the example annotation shows. It would be helpful to see how the agent’s performance relates to how captions match the agent’s task. 
  2) the target object needs to be in the background (see Questions), it is unclear how many times the agent needs to see the object in the background in order to interact with it
questions: - In the experiment of section 3.3, the authors only use the drum data to train the caption model and then use this caption model to train the agent without drum interaction data. In the second training stage, does the agent see and caption the drum? If not, it is unlikely the agent can learn to interact with this new object. How much interaction data in this experiment contains a drum as a background object? This would be a key contributor to the zero-shot performance.
- Do the captions cover most of the visible objects in the scene? The proposed approach will perform better if the caption has higher coverage of objects. How robust is the proposed model when having lower object coverage?
limitations: - As the authors suggest, the proposed intra-agent speech can only generalize to object identification tasks as the captions and observations are visual and object centric.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors model intra-agent speech as latent codes for embodied agents. In optimizing the ELBO, the authors use a frozen language model to enforce a nautal language like latent code distribution. They test this model on image caption tasks with both paired data and unpaired data, and find with the same amount of unpaired data, semi-supervised could benefit from unpaired data. They also find that the model can learn to speak about objects that are rare in the paired data and only passively observed in the unpaired data. Finally, they find that just through adding “drum” related captioning data, the agent can learn to lift drum. 

strengths_and_weaknesses: Strengths

The intra-agent speech concept has rarely been discussed in the embodied agent literature. 
Their finding about generalizing to zero-shot interaction tasks from captioning data is very interesting. 

Weakness:

Using variational methods for semi-supervised learning is not a new idea. The first finding on the benefit of unpaired data is expected. This may not necessarily be a weakness, but branding it as “intra-agent speech” might be misleading and hyping. Since there’s no evidence that this is similar to the human inner speech, it should be made clear that this work has nothing to do with inner speech, and is just a latent variable model. My scores are based on the assumption that the authors could make the amendment to make this clear. 

questions: Most of the zero-shot results are only on a single concept -- drum. What are the obstacles preventing the authors from conducting the experiments on other concepts as well? 
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper is a novel proof-of-concept of embodied agents gaining the ability of new tasks with two kinds of semi-supervised learning mechanisms, namely 'intra-agent speech'. The results are three-fold - embodied interaction of agents within a simulated environment (or intra-agent speech of unified agent): (1) enhances the data efficiency of the image captioning subsystem of the agent, (2) reduces the number of data required to learn about the new object, (3) allows motor task involving the new object.
strengths_and_weaknesses: Strengths
- This work largely unifies the frameworks from image captioning, image generation, and reinforcement learning to show the newly devised concept 'intra-agent speech' can work as an effective semi-supervised learning mechanism.
- The writing is clear enough for readers to understand what hypotheses are established and how they are tested.

Weaknesses
- The conclusions derived from the experiments aren't backed with enough numerical details. For instance, how should the readers conclude that the proposed methods allow a model to 'quickly' learn with 'little' supervision? To meet a broader audience, those ambiguities should be resolved.
- The zero-shot skill acquisition task (section 3.3) is only experimented with a single specific object (drum), which is not convincing enough.
- 95% confidence intervals are obtained with only 3 experiments, which does not seem to be enough sample size. The reviewer considers 5 as a bare minimum for such statistics. (It's also mistyped - "confident interval" should be "confidence interval")
questions: - Where can I find the image captioning performance of the model trained with only semi-supervised pretraining (i.e., intra-agent speech without an agent)?
- What does the 'indicator embedding' in line 151 refer to?
limitations: The reviewer is unsure whether this work shows that the inner speech of the agents is helpful in efficiently learning about the world as humans do. Unless there is a sufficient amount of experiments and literature in a similar field, such over-claiming should be avoided.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors present an "intra-agent speech" framework combining a semi-supervised image captioning model (herein M1) and an agent grounded in the Playhouse environment which takes action and answers questions (herein M2). M1 is trained first on a small amount of labeled data in the Playhouse domain, and its outputs on novel Playhouse samples are used to provide supervision to M2 -- an implementation of the "intra-agent speech" idea. The authors present results on 1) M1's semi-supervised captioning data efficiency, 2) M1's low-shot captioning data efficiency for novel objects, and 3) M2's low-shot performance for novel objects. They argue that this implementation of intra-agent speech provides data-efficient supervision to M2 for both language use and acting in the world.
strengths_and_weaknesses: Overall, the paper presents an interesting idea motivated by a psychological concept, but the execution and presentation are far from thorough. The model design and evaluation seem to contain several arbitrary choices, and little analysis in the paper helps the reader understand *why* the model improves concretely.

I do not follow this subfield closely and can't speak to the originality or significance of this paper with confidence. I do regard it as significant in that it makes a good-faith attempt to implement a psychologically motivated concept.
questions: 1. **Model design:** What is the motivation for your intra-agent speech implementation? A more obvious setup (to me) would be to have the outputs of a captioning model directly affect the representation of the agent, rather than serving as supervision in an auxiliary task. This would seem to resemble more closely the "intra-agent speech" idea -- in that the inner language is part of / directly influences mental representations online. While I'm not so attached to this alternate implementation, it is important for me to know why you chose your particular implementation. Can you explain why you chose this auxiliary task route? Why might it be the most effective and/or cognitively faithful method for implementing intra-agent speech?
	1. For what it's worth, I'd strongly prefer a thorough discussion on this design choice and alternatives in place of the ~1.5 pages dedicated to the generative vs. contrastive training setup.
2. **3.1 evaluation:** Why is "color-object accuracy" the right custom metric to develop here? I have many other more stringent questions I'd want to ask about a captioning model in this domain. For example:
	1. Of the concrete nouns mentioned in a caption, how many have referents in the scene?
	2. For noun phrases "the X," how often is there a unique referent selected by X in the scene?
3. **3.2 training:** What is the training method here? Is the model being fine-tuned with drum examples, or retrained afresh with a dataset that is the union of the original dataset with some fixed number of drum examples?
	1. Possibly related question -- what happens to the log-probability of captions for non-drum images as the drum-training proceeds?
4. **3.2 evaluation:**
	1. What is the ideal "drum true positive rate?" Captions needn't include every object in a scene. / Humans won't always mention a drum when it appears in an image. Why is 70% better than 20%?
	2. The drum false positive rate result doesn't look so interesting to me. There may not be any significant differences between semi-supervised and supervised training. I'd rather see here the other conditional P(drum present | drum mentioned).
5. **3.3 evaluation:**
	1. Over what trajectories are human reward labels collected? Does it include ground-truth captions and ground-truth motions? This seems very important for making sense of the absolute value of the rewards shown in figure 6.
	2. Can you show the full distribution of human reward labels, without normalization? (At least in an appendix.)
	3. How does the "ask color" task work? This doesn't seem analogous to the captioning task imposed on the agent during training.
6. **Arbitrary evaluation choices:**
	1. Why drums? Do you plan to repeat the analyses for other objects? You could have a leave-one-out style evaluation for all objects that appear (or appear with low frequency, at least) in the unlabeled data.
	2. Why just lifting? How many motion tasks are there available to test?
		1. What about a task that is more semantically complicated (e.g. "put the drum on the table") or motorically complicated (e.g. "balance the drum on its side")?
7. **Interpretation:** Why does this work? I think the paper would be much more compelling if the authors could provide detail (through a model comparison, representational analyses, or error analysis, for example) on what the auxiliary captioning objective (AKA intra-agent speech) is doing for the model. The particularly interesting result is that the captioning objective affects not just the linguistic output of the agent but also its motion behavior.
limitations: Yes.
review_title: null

Meta-review: metareview: This work aims to provide a faithful translation of "inner speech" to a virtual agent via image captioning. This is integrated via an auxiliary loss and shown to assist in zero-shot transfer to a new object.  This aids in reasoning about the otherwise overly rich and potentially spurious visual space. 

The good faith approach and efficient learning results were both viewed as strong positives.  The primary concerns revolve around the very limited number of samples and whether the approach can actually generalize beyond the limited evaluation domain presented here.  The integration of additional experiments promised will dramatically strengthen the work
confidence: Certain
Decision: Accept