Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper lies in the space of using non-Markov reward functions to model temporal task, and in addition to that attempts to generalize instruction following beyond single instruction provided in a formal language. The authors propose a dual network solution to embed the environment representation and separately a goal specific reasoner grounded within the environment. For planning with temporal formulas, the authors extend prior work on Task temporal logic. Although the algorithm for this remains largely unchanged from prior work (please clarify if this is wrong).

The results demonstrate that the latent goal representation helps in generalizing at both the object level and at the task level. However the presentation of the results can be significantly improved (please refer to the main review).
main_review: **Major Comments**

*Sound policy network design choices:* The design choices in the proposed latent goal representations appear to be sound. It appears that joint training of the goal specific information with the task embedding is beneficial to generalization. And this design choice appears to be benefiting the performance irrespective of the network architecture of the central module. However, this is hard to judge considering the presentation of the results.

*Training and testing on multiple formulas:* I like the fact that the authors have sampled multiple instructions to test on and not on a few handpicked formulas. The authors should provide the definition of the sampling algorithm for the instructions in the appendix.

*Presentation of the results:* My biggest issue with the paper is how the results are presented and analyzed. I urge the authors to use bar charts instead of raw numbers to present the results as the comparisons are much easier for the readers. Further I believe that the standard errors of many of the conditions overlap with each other. The authors should perform statistical analysis appropriately before claiming superiority of any of the conditions over the others. With so many experiments, and data points, sound statistical analysis is important. a 2-way ANOVA (independent variables being network architecture, and latent goal representation or the classical architecture) with repeated measures over the minigrid and minecraft domain seem to be the appropriate statistical approach. Without this, the claims of benefits of one over the other are tenuous. While the authors can include this in the final version, I believe the authors should be able to run this before the responses are due.

**Minor comments:**
Please consider the following suggestions for clarity:
1. I would suggest the authors to bulletize the baseline architectures, instead of writing a long paragraph. That would make it easier to parse the important features. 
2. Separate discussion of the results from the ablation studies, right now you only get to the ablations at the end of that subsection
3. I would have liked to see some contrastive description of SATTL with respect to TTL. How does SATTL improve the expressivity of TTL? What types of task cannot be expressed in TTL but can be in SATTL?



summary_of_the_review: I believe that as written the paper is not ready for publication. I believe that statistical analysis of the results is necessary, and can be reasonably completed in the response period. I would be willing to upgrade my scores (correctness and significance) based on that. I also believe that the edits for clarity would significantly strengthen the paper.

Update: The authors incorporated most of the comments, and the statistical analysis bears out the major claims.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work focuses on training (via deep RL) task conditioned agents for a given class of workspaces. Tasks are formalized as a variant of finite temporal logic (and thus encode a regular language). Tasks are either provided as text or via visual diagrams. A key goal of this work is to perform well on tasks that lie outside of the training distribution. The paper illustrates on two common grid like domains with the results of a number of various architectures and ablations reported.

The approach taken is an adaptation of deep reinforcement learning, with the primary contribution being a slight, but seemingly important, modification of an existing architecture to create a separate embedding for (i) a task agnostic state encoding and (ii) create a task specific state encoding. Furthermore, in order to encourage the network to generalize between tasks, an information bottleneck is placed on the task specific encoding.
main_review: In my opinion, this paper is well laid out and tackles the important problem of creating agents that can perform high level temporal tasks. While a fairly incremental contribution, the thorough empirical analysis indicates that performing out of distribution temporal tasks is possible.

One thing I was surprised was not explicitly done was to use the underlying automata structure of proposed reward function to further accelerate the RL loop. In particular, I recent work on reward machines (which can encode regular languages) uses a trick similar to hindsight experience replay to provide counterfactual examples that and mitigate reward sparsity issues. Looking at the training plots in the appendix, I wonder if the generalization may have occurred if many fewer steps. I am also curious if such a trick could be used to remove the small (and seemingly adhoc) penalization incurred at each time step (which is in theory redundant with the discount rate, but a source of possible reward bugs).
summary_of_the_review: Overall, while somewhat incremental, I think this paper adds a valuable contribution, is very thorough, and pretty coherent. The focus on simply modifying the inductive biases in the architecture (rather than dramatically changing training) is also a good venue fit.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The manuscript proposes 'latent goal architectures' for following formal instructions in OOD scenarios.
main_review: Section 3.2: This approach, especially with the definition of the symbolic module, seems to bypass many of the challenging problems in instruction-following: sparse rewards (e.g., rewards only available at the end of the episode), with no explicit progress-monitoring provided, noisy instructions and environment, no basis for task-specific reward-shaping, goals are not necessarily conditioned on the agent’s current state (i.e., leading to issues with initial heading, localisation, etc.). The SM provides a lot of privileged information, through its labeling function and comprehensive reward function.

Section 3.2: How is ambiguity handled — i.e., when there exists multiple ways that T can be entailed?

Section 3.3: The manuscript states "Consequently, it can be beneficial for agents to compute both human instructions and sensory inputs in a dedicated channel, generating a latent representation of the current goal. We call this type of deep learning model a latent-goal architecture." By itself, this is not novel: there are a plethora of works in learning robot skills, and they have dramatically less supervision.

Section 4: Table 1: Train/Test results should be compared separately. Bolding schemes (to highlight the best-performing model) change, from table to table; this should be made consistent.

Section 4: The manuscript, which pursues so-called 'latent goal architectures' for following formal instructions, somehow makes reference to neither the robot skills literature nor the robot instruction-following literature. Additional experiments and comparisons are needed.

Throughout: The manuscript relies on the assumption that "human instructions can be separated from the sensory input". This is a strong and unrealistic assumption, in practice. What need to we have, then, for multiple sensor modalities, when the instruction (whether natural or formal language) can completely specify the task?
summary_of_the_review: I have some issues with the novelty of the work presented, the accuracy of the claims made, and the somewhat lacking comparison with the related work and key ablation experiments. See main review, above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces an neural architecture for a DRL agent tasked with learning to solve temporal logic specifications. Results show the proposed architecture outperforms existing ones when it comes to performance on instruction following in unseen environments.The proposed architecture takes in an observation and a task description, which appears to be transformed into a reward function that is then optimised. The key contribution appears to be around the inclusion of some latent state information about the environment in the reward specification phase, with the motivation that this makes it easier to interpret a human temporal logic specification of a task.

Results appear to be strong, and do show that this results in substantial performance increases in unseen tasks.

Unfortunately though, I found this paper a difficult read and in need of substantial work before it is ready for publication, primarily around the problem framing, method description, discussion and motivation. At present, it lacks the clarity required to best showcase the results, and I cannot confidently say I understand this paper, because it is not explained or presented clearly enough.
main_review: Strengths:

It seems like a good idea to include environmental information in the reward specification phase, and empirical results do show that this is effective.

Weaknesses:

At present the paper suffers from severe clarity issues, which make it difficult to evaluate. Specifically:

- The abstract doesn't actually explain the task or method, and reads more as a collection of jargon. This needs to be reworked to more clearly specify the problem formulation, goal and proposed solution.

- Contributions need to be made clearer

- Problem formulation: the paper is in need of a clear problem formulation early on, explain exactly what the task is, inputs and outputs, and key steps in the solution, early on. Right now, the background provides an MDP definition that could be repurposed to explain the task, and motivate the key reasons for the proposed framework much earlier on.

- SATTL section: it's unclear what this adds to the paper - right now this reads as a floating intro to SATTL that just breaks the flow of the text. I'd move this to an appendix, or rework to explain only what is required from the perspective of reward specification.

- I only started to understand the paper in section 3.3 - I'd consider moving this much earlier, and motivating the problem using this example.

- It's difficult to map the framework in 3.2 onto the architecture in Fig 1. The method description needs a lot of work.
-Fig 1 and the associated discussion is very confusing. Right now, I'm unsure if the proposed solution only operates in the reward specification phase, and is then used to optimise an actor-critic policy, or if something interesting happens in the latent state of an actor critic network, or if a meta-agent is being trained to predict a zero-shot value function and policy. 

Overall, I believe this paper would benefit greatly from being re-read from the perspective of someone not involved in the project, because at present it does not adequately communicate the work completed.
summary_of_the_review: I think there is interesting work in here, and a nice story to be told. Unfortunately, the current structure of the paper is not good enough to support this, and the paper is in need of a substantial amount of rework to improve it's clarity.

At present, it lacks the clarity required to best showcase the results, and I cannot confidently say I understand this paper, because it is not explained or presented clearly enough.

==== Post rebuttal comments ====
Thank you for revising this work, I think it is much stronger now after the restructuring and clarification, so have increased my score.
review_title: null

Meta-review: The paper considers the problem of learning to carry out novel, multi-task instructions specified via temporal logic using deep reinforcement learning. A specific focus of the paper is improving generalization to test-time instructions that differ from those encountered during training. To facilitate this generalization, the proposed architecture encodes a latent specification of the goal according to the given instruction and environment state that is then combined with a task-agnostic environment embedding. Experiments on grid-like domains demonstrate that the proposed framework outperforms recent deep RL approaches to satisfying temporal logic-based instructions.

The instruction-following problem has long been of interest in the robotics, ML, and broader AI communities dating back several decades. The problem has received renewed attention in the last few years, largely as a target for neural network-based multi-view and RL learning architectures. The primary contribution of this paper is the proposed extension of existing deep RL approaches to reason over a learned, latent goal specification as a means of improving generalization to novel test-time utterances. The approach is sound and several reviewers agree that the ablation studies together with comparisons to contemporary deep RL architectures support the advantage of these inductive biases. The reviewers raised initial concerns regarding the statistical significance of the results and the clarity of the presentation. The authors provided detailed feedback to the reviewers and updated the paper to address many of these concerns, largely satisfying two of the reviewers.

However, concerns remain that the paper doesn't adequately position this work in the context of the decades worth of research in instruction-following. Early work in this area focused on interpreting highly structured instructions (e.g., formal logic-based), first using rule-based methods, and then parsers trained via supervised learning. Over the past decade, however, the field has largely moved towards learning to follow instructions conveyed in "natural" language, which brings with it a significant number of challenges, including the assumption that test-time instructions will inherently be out-of-distribution. That is not to say that the contributions of the paper aren't interesting---they are, but in the relatively narrow scope of deep RL-based approaches to following structured, temporal logic instructions.
Decision: Accept (Poster)