Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Authors address a class of RL challenges where the main task implicitly consists of a sequence of subtasks, and the main goal of an agent can be split into two parts: to decompose the tasks into a sequence of subtasks (planner module) and to perform each subtask individually. They consider a scenario where the current state (map of the world) is not fully observable. The key idea of this paper is to use a model of the map to supply a planner module with a hypothetical map of the world so that it can devise an on-average well-working plan. The authors applied their method to several environments including a 2D Minecraft-inspired environment and show that it is able to outperform prior methods. Their method eliminates the need for the human user to produce a demo solution of the ultimate goal but still requires the user to specify the logical predicates of the ultimate goal and goals of micro-tasks that serve as milestones in planning. These predicates are based on some DSL also specified by the user.

Although this paper has some weaknesses (see below), it makes a step towards a promising direction, so I slightly lean towards accepting.
main_review: Overview.
_____

The proposed three-piece approach helps RL agents in goal-oriented partially observed environments to complete complex multi-step tasks. Unlike other methods, which require that solution strategies are supplied, the approach needs only the specifications of sub-goals and the ultimate goal from the user.

A quite peculiarly named `hallucinator` piece is a conditional generative world model (C-VAE) trained on random exploration data, to predict realizations of unobserved sectors of the world map (in terms of their features) and hallucinate objects located therein, given the observations made so far.

The program `synthesizer` piece samples possible worlds conditional on the current observation history and applies a MaxSAT solver to compose a shortest program, i.e. a series of "components" (milestone micro-tasks) that guides the agent to the ultimate goal.

The final piece is the "executor", which traces the synthesized program by executing RL options associated with each "component" micro-task. The hallucinatory program synthesis step is repeated, should the agent fail to complete the goal or halt on running out of the allotted time, this time, however, leveraging newly collected observations.

Each option, i.e. a policy with a particular termination criterion, is pre-trained in the environment to complete the user-specified micro-task. For speedup, the rollouts during option pre-training are collected by leveraging the "ground truth program synthesized based on the fully observed [version of the] environment" (L217) for the ultimate goal.

Strong points.
_____

Like its predecessors Sun et al. (2020) and Andreas et al. (2017), the method produces a nice inductive hierarchical structure of goals and subgoals. A policy option is learnt for each specified zero-tier micro-task in the environment, and then the program synthesizer creates a script for a specified higher-tier task by composing lower-tier tasks and packaging this into an associated composite option.

This paper's method itself completes an important `micro-task` on the path to program-guided exploration, preferably synthesized on-the-fly, which would be capable of generating milestones from prior knowledge and micro-tasks, or a curiosity-based point of Interest detection method, that produces such micro-tasks for which the options are also learnt on-the-go.

Weak points / criticism.
______


1)	Experiments are set on some tasks composed especially the particular paper, and this is not obvious what should be SOTA for them. Also, five replications of each experiment seems to be on the side of being too few.

2)	Exploration: the proposed method of exploration is exploitative (goal-oriented) and do not have any curiosity features. Such approach seems to be limited to the environments with finitely explorable map. It is unlikely that their method would be successful for such challenges like obstacle tower challenge (Unity Technologies).

3)	Some important details of the paper are not clear:
- What maps S (should be) used to sample training data for a VAE hallucinator? Can we make sure that the (information from the) maps where the agents are tested do not leak to that training data for VAE?
- A connected question is: what does mean “the actual distribution P(s\mid o)”, how can it be defined and how can we sample it?
- What is “a randomly generated map” (L99). What is a generative model?

4) Although the proposed program synthesis approach is interesting, I am concerned that the variability and feature simplicity of the 2d-craft environment have played a significant role in the success of the presented experimental results. Indeed, the pre-trained micro-task options could memoize the peculiarities of this procedurally generated environment.

5) Another limitation of the particular instance of the presented method is the need for the full state to be accessible (or relevant features thereof). There are environments, e.g. the [NetHack](https://arxiv.org/abs/2006.13760), which have feature-rich observation space and very long branching exploration paths, and the full state of the “world” is unavailable, and has to be discovered. In this situation the hallucinator, which is trained on random exploration, will be extremely unlikely to ever stumble upon the ultimate goal, let alone be able to reconstruct the entire state. It will face the same challenge with feature-rich states, so there is a need to devise a method to limit the scope of the generative model to only the likely relevant locations.


Corrections, typos and RFCs
_______

- L291: It is unclear what is meant by “random hallucinator. Is it untrained, is not conditioned on the observations at all, or not updated with new exploration information?
- L290: It is also unclear what is meant by a synthesizer that makes “the best-case assumptions on about the unobserved parts” on?
- L107: “final state s_{-}” should be “final state s_{+}”?
- L 153: the conjunction sign should be rather conjunction?
- L35: “the agent automatically [sythesizeS]”
- according to NeurIPS style the captions should precede the tables.
- figure 6 should be table 2.
- increasing DPI on the included figures would enhance their accessibility


__________
After rebuttal

Summing up, in order to automatically find a program, which achieves the main goal of the agent, the MPPS method requires that we

* know the predicate specifications of micro-tasks (options) and the main goal;
* have well pre-trained policies that can explore well and complete micro-goals (options);
* have a generative model that reconstructs abstract state descriptions from partially observed data

The main contribution is an approach to automatically synthesize guiding programs for reinforcement learning policies under partial observability.

Having read all discussion, I slightly lower my score (6 -> 5). On one hand, the idea is promising, the direction seems important, and the authors addressed many important questions. On the other hand, I agree with kENU that many important details regarding training, ablations, and baselines are needed in the paper. Further, as detailed below, I think the paper needs additional experiments. The proposal about “interweaved hallucinator training with executor learning” should be discussed in the paper.

Now, although I am convinced that pre-training the component options under full observability is a way too privileged in the partially observed settings, I believe that such privileged setting still can give valuable proof-of-concept results for the core method: program synthesis with a help of a generative model for states. The availability of well-trained options can be considered as an assumption for the applicability of the method, and I believe that this assumption is reasonable for future RL systems, which will share options among tasks and environments.

However, I have additional concerns about baselines. Currently, the End-to-end baseline, the one with many micro-actor DNNs, is severely handicapped: (1) it does not have access to a hallucinator (mentioned in the response to Wt3P, `experiments/craft_nn.yaml#L7`, and `models/simple_nn.py#L59-66`), and (2) it is not recurrent (judging from a quick regex over the supplied codebase), contrary to the provided response to kENU. Point (2) is rather concerning since it is unclear how the non-markovian dependence on the full history is handled.

At the same time, the World Model baseline (recurrent) does not have access to pre-trained options, which makes the comparison unfair. Thus, the baselines do not allow us to separate the effect of good micro-goal options from the synergy between the hallucinator and the synthesizer.

I want to reiterate: in the codebase the ”hallucinator” is a CVAE with a Bernoulli decoder (`models/cvae.py#L122-124` and `train_cvae.py#L198`) trained on random rollouts (`train_cvae.py#L418-420`) to reconstruct full one-hot encoded (`train_cvae.py#L395-411`) abstract states (`environment/box.py#L444-470` and `environment/craft.py#L953-990`) from partial observations overlain on an empty binary grid map (`environment/box.py#L542-615` and `environment/craft.py#L953-990`).

In light of the claimed benefits of synthesized programs, I think a good baseline/ablation experiment would be to replace the synthesizer with a recurrent neural policy on pre-trained options, which __also has access__ to the hallucinator or an equivalent world model (also pre-trained on a fully observable environment). To fairly judge the benefit of program synthesis here it may be necessary to compare the number of requests each approach makes to the world model.

As for the claims of reduced user supervision, I suspect that identifying micro-goals and specifying them as logical predicates still demands that the user thoroughly analyzes the environment (L32). So the toning down of the contributions of the paper seems appropriate. Besides, micro-goals' logical predicates as they are described in the paper do not seem to be very high-level (L35): one has to tinker with the before- and after- states, enumerate positions, and zones in the grid, and compose conjunctive forms (L496). Despite this, I am inclined to agree that such specifications are more basic than a program description of an optimal solution as in Sun et al. (2017).
limitations_and_societal_impact: I do not see any potential negative societal impact
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a neurosymbolic, hierarchical agent architecture which uses program synthesis at the top level of abstraction to generate a symbolic plan. The plan is executed using learned options, which are trained via supervision. To address partial observability, a generative model of symbolic states given observations is trained and used to sample possible states of the world, which the program synthesizer optimizes with respect to. Replanning (i.e. re-running the synthesizer) is performed every so often in order to update the plan/program with newly observed information.

I really like the idea of learning a generative model over states and performing program synthesis with respect to that. This idea has close ties to other approximately Bayes-optimal methods for handling partial observability, and seeing it successfully instantiated in the program synthesis setting is an important contribution. More generally, the topic of the paper is timely and important, as program synthesis and neurosymbolic methods more generally are an important area of current AI research.

However, my sense is that this paper is not ready for publication at NeurIPS. This is primarily due to (1) a lack of clarity and (2) inappropriate and poorly-tuned baselines; secondarily, I am also concerned by the (3) overreliance on privileged information, and (4) lack of grounding in the automated planning / TAMP literature (though to a lesser degree than points 1 and 2).
main_review: **Clarity**

The paper suffers from a major lack of clarity. I had to re-read the methods multiple times before I felt like I understood what was really being done, and I’m still confused about many things. There are many details missing which make it hard to understand exactly what experiments were run, how they were evaluated, how the baselines are implemented, and so on. I list below some things that particularly confused me, but as a more general recommendation, I would suggest including much more information in the appendix and referring to it liberally. There should be enough information in the paper for someone to be able to reimplement it in entirety, and that is currently far from the case.

* What exactly is included in the observations that the hallucinator is trained on? The paper says “we assume the observation o on the current step already encodes all observations so far”. One part of the paper led me to believe that o was the count of state features (e.g. the number of gems), rather than the full map itself. But after a while longer of combing through the paper, now I’m back to thinking that observation is the more typical use of the word (i.e. the map). But how is this encoded? Is it an image? Is it a more symbolic representation? Also, what does “all observations so far” mean? Is that a matrix of observations from all previous timesteps? Or do you just mean you assume the Markov property here?
* It sounds like the hallucinator is trained using ground-truth state information, but this is not explicitly stated. Is this true? If not, how do you deal with the fact that a random policy is unlikely to be able to actually observe the full state (since for long-horizon tasks this will be exponentially unlikely)?
* It’s not clear to me if the hallucinator and executor are trained simultaneously or not, and it’s not clear what the x-axis of Figure 3 shows. Are those episodes required by the executor? Does it include the data required to train the hallucinator, too?
* There are almost no details given about the baselines, which makes it hard to evaluate whether they are appropriate. For example, with the End-to-End baseline, the paper states there is “one DNN policy per goal”. What does this mean? How are the separate policies trained? How many policies does that amount to? 
* What are the goal specifications that the agent is evaluated on? In your results, are you averaging across different types of goals? Are these evaluated in equal proportion? On line 230, the paper says the goal can be “one of 10 possibilities” but as far as I can tell it is not stated anywhere what these possibilities are. It would help to include this type of detail in the appendix.
* Do all agents (including baselines) get to see the fully-observed environment during training? The paper states that “just for training [the executor], we use the ground truth program p synthesized based on the fully observed environment (since we can explore the entire map and post-hoc generate p)”. However, this seems to me to put the program synthesis agent at a very unfair advantage compared to the other agents, which presumably don’t get to see the fully-observed environment during training in any capacity. The other agents should either also get to see the fully-observed environment, or you need to explicitly include the data cost for exploring the entire map to generate p. (It’s also not clear to me that all maps can even be fully explored in a reasonable amount of time, given that part of the map may be blocked off by rocks/water).

**Quality**

There are two issues that pertain to quality: baselines and privileged information.

First, the baselines seem poorly chosen and poorly tuned. Given the main contribution of the paper is a method for dealing with partial observability via program synthesis, it would seem to me the more appropriate comparisons would be with either (1) other approaches to program synthesis that don’t account for partial observability or (2) neural approaches that do account for partial observability. However, the paper only compares to neural approaches that don’t deal with partial observability, which seems inappropriate.

But assuming the choice of baselines themselves is ok, I had a lot of confusion about how they were even implemented (see above), which makes it hard for me to evaluate whether they are fair. From what I was able to gather, I get the impression that they are not. For example, the paper says on Line 266 that “We use the same actor (resp., critic) network architecture 267 across all approaches—i.e., an MLP with 128 (resp., 32) hidden units”, but this seems like a very poor baseline: it is probably not deep enough (the number of layers is not stated, but I’m assuming just 1?), 32 hidden units is almost certainly too small for the critic, and the architecture should probably be a CNN, not an MLP, as MLPs typically do not perform very well in visual/grid-world settings. Similarly, for Box World, where a CNN is used, the paper states that only one 3x3 conv layer is used, which is also probably not the best choice. Why was a different architecture chosen than what was used in the RRL paper (which is two 2x2 convs followed by ReLU)? It is suspicious to me that the relational approach does *worse* on Box World than the other neural approaches, which is the reverse of what was found in the RRL paper. Including experiments in the appendix indicating that these baselines are correctly implemented (e.g., that they succeed at the fully-observed version of the task) would help me have more confidence that they aren’t a strawman comparison.

Second, the method relies on a lot of privileged information and domain knowledge. Specifically, from what I could tell, the method requires: a DSL, counts of state features (which may include information above and beyond the DSL, e.g. zones, boundaries, etc.), transition operators (referred to as “components” in the paper), a logical goal specification, and access to the fully-observed environment during training. I’m certainly willing to allow for some amount of this; I think it’s hard to make progress with program synthesis and other symbolic approaches without at least allowing access to a DSL (and in much of the TAMP literature, transition operators and goal specifications are also given). But the choice that pushes it over the threshold for me, at least, is to allow the agent to see the fully-observed environment during training, which really just seems like it defeats the purpose. Isn’t the whole point of the paper to have a learning algorithm which can deal with partial observability? 

**Originality & Significance**

As far as I am aware, applying the Thompson sampling technique to program synthesis in order to account for partial observability hasn’t been explored. As I said previously, I think this is a great idea and it’s important to demonstrate how to get it to work in practice.

However, the paper tries to lay claim to another contribution which I don’t think is justified: ”the user only needs to provide a high-level specification for the goal of the task; then, the agent automatically synthesize a suitable program that is used to guide the policy” (Line 34). In particular, the method is not really doing program synthesis but rather task planning—which is an area that has been highly studied in the context of task and motion planning (TAMP). Like the proposed method, TAMP relies on having access to the DSL, goal specification, and transition operators. Yet, TAMP isn’t even mentioned in the paper at all. Perhaps the authors aren’t aware of this area of research, in which case, here is a recent review of such methods [1]. Moreover, while TAMP is not my area of expertise, from a quick literature search, I found a few papers that propose to deal with partial observability in TAMP [2, 3]. I think that TAMP at least deserves a mention in the related work section, and I think that the paper should tone down its claims about the program synthesis in and of itself being a novel contribution.

**Minor comments:**

* Typically when comparing learning curves of RL agents, the x-axis should report the number of environment transitions, not the number of episodes.
* What does the *1000 mean in Figure 3? Is that indicating that the x-axis actually goes from 0 to 200k or 400k episodes?
* “We leverage program synthesis [27] with the world models approach [9] to address these issues” → this statement isn’t really accurate, your generative model of states given observations isn’t really what world models does. I would recommend being more precise and just saying you train a generative model over states given observations.
* The approach of sampling a realization of an MDP and planning optimally with respect to it—i.e. essentially the Thompson sampling approach—is approximately Bayes optimal in some settings, e.g. see [4]. Your approach is quite similar and it might be worth discussing this.
* Figure 7 is very hard for me to interpret. I would suggest not having two scales on the same plot but instead plot the variance as error bars around the mean.
* “Significantly” is used a number of times throughout the paper to indicate a substantial effect. However, “significant” has a precise statistical meaning, which is that we can be confident one value is higher than another. Since you perform no statistical tests in the paper, I would recommend avoiding the use of the term “significant” and stick to more general terms like “much higher/lower” or “substantially”.

[1] Garrett, C. R., Chitnis, R., Holladay, R., Kim, B., Silver, T., Kaelbling, L. P., & Lozano-Pérez, T. (2021). Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4, 265-293.
[2] Phiquepal, C., & Toussaint, M. (2019, May). Combined task and motion planning under partial observability: An optimization-based approach. In 2019 International Conference on Robotics and Automation (ICRA) (pp. 9000-9006). IEEE.
[3] Kaelbling, L. P., & Lozano-Pérez, T. (2013). Integrated task and motion planning in belief space. The International Journal of Robotics Research, 32(9-10), 1194-1227.
[4] Guez, Silver, & Dayan (2013) “Scalable and Efficient Bayes-Adaptive Reinforcement Learning Based on Monte-Carlo Tree Search”
limitations_and_societal_impact: I have discussed a number of limitations above, such as the assumption of full observability during training. There are a few other less important limitations as well. For example, environments must be static (i.e. once something is observed, it is assumed to not change unless it is within the agent’s field of view), and actions are deterministic. The proposed method would not work well if either of these assumptions were violated. I don’t necessarily expect the paper to address this, because it can’t be expected to solve all problems at once, but these limitations should be more explicitly called out.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors introduce MPPS, which stands for model predictive program synthesis, a method to leverage programs to guide reinforcement learning in partially observable problems that require planning over long-horizon. MPPS  trains a generative model to predict the unobserved portions of the world, and then synthesizes a program based on samples from this model in a way that is robust to its uncertainty. The authors illustrate the performance of their algorithm on a variety of benchmarks including a 2D Minecraft-inspired environment.
main_review: I warmly thank the authors for their work.

I think that the domain covered by this paper is very exciting and that this type of approach is a very promising direction for Reinforcement Learning. However, in this current form this paper exhibits several limitations:

- This paper was very difficult for me to follow. I think it would require more care to introduce the concepts, position the method with respect to the literature, and better detail the examples.

- I did not get in the introduction what is precisely the problem that the authors aim to solve and why the current methods fail to do so. 

- At the very beginning of the paper, both concepts Domain-specific language and Task-specific program remain unclear to me. Especially, the example “gather wood” is used for both.

- The authors often mention that their method alleviates the user burden to provide a guiding program for every new task. What the authors mean here remains unclear to me and I believe an example would help to better understand.

- Many concepts are mentioned without being introduced first. For instance, in the very beginning, the authors talk about options without explaining what options are.

- Many mathematical notations are also not properly introduced.  For instance, on page 2 in the definition of the executor, p = c 1 ; ...; c k, the variables c_1, …, c_k are not introduced and I do not understand what they refer to.

- Section 2 covers a motivating example that remains unclear to me. I think that this section is very abstract for a motivating example and that most of the mathematical notations might be kept for later in the paper and replaced here by concrete examples.

- In the related work, the authors missed the literature about Neural Programmer Interpreters and notably AlphaNPI that is also an RL method guided by programs that would be relevant as a baseline in this study.

- The authors mention several times options but do not cover them properly in the related nor compare to them. I think that at least the Option-Critic and HAC should be mentioned, and even compared to, in this study. I understand the point made by the authors that work over options does not cover the partially observed setting however, a simple baseline to validate that this method is more relevant than using options might be to consider an option algorithm such as HAC and add a recurrent core to the high-level policy to handle partial observability.

- The different neural networks architecture as well as the RL agent used to train MPPS remain unclear to me. I think that the paper lacks a section that carefully details these points to enable reproducibility.

- The experiments and the environments considered seem impressive but should be better introduced. The authors might dedicate more space to explain what is the observation space, action space, and reward function. For instance, the Ant-Craft environment is very briefly introduced and it remains unclear to me what it really consists of.

- The authors compare their methods only to ablations. It would be important to consider baselines such as the ones mentioned in the previous points (AlphaNPI, Options method, …).
limitations_and_societal_impact: This method does not add limitations or potential negative societal impact to existing reinforcement learning methods.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The algorithm is set in a problem where the grid for exploration is not fully observable. The authors introduce a new approach called model predictive program synthesis (MPPS) that involves a hallucinator, a synthesizer and an executor. The hallucinator predicts the state features with CVAE, the synthesizer provides programs through optimization of a MAXSAT problem. And the executor executes the programs. 

main_review: The paper is shown to outperforms the other comparison methods. However, I think it’s missing some discussions and comparisons that help put the problem and method in the bigger context. 

I’d like to see a bit more discussion on why this particular model is able to outperforms the other methods. Is it because of the SAT solver? or the hallucinator? Or because it didn’t specify goals? A more in-depth discussion would add more weight and clarity to the paper.

How a hallucinator is trained is not very clear to me. A diagram or an example input/output would be nice. This would facilitate understanding of the method. 

Explanation of the end-to-end method is not clear. Does it also have a hallucinator? 

It seems like some of the maze or corridor environments in deep rl also have the same problem of not having full observability. There are already solutions exist for those types of problems without program guidance. Can you explain why program synthesis is necessary for this type of problems? Or at least are there any quantitative comparisons that show improvements or such? 

limitations_and_societal_impact: I don’t think there are any ethical concerns regarding this paper. And some limitations are also addressed. For specific problems, please refer to the last section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper aims to address the problem of solving long-horizon tasks in a partially observable environment by introducing a program-guided reinforcement learning framework. Specifically, given a task specification, this paper proposes to first synthesize a high-level instruction (i.e. a program) that consists of a sequence of subgoals and then execute the instruction to fulfill the task. To this end, the paper proposes a framework that comprises three components: (1) a generative model that can hallucinate the unobserved portions of the environment, (2) a program synthesizer that can produce a program that would fulfill the task specification, and (3) a program executor that can execute the program. The experiments show that the proposed framework outperforms the baselines including an end-to-end learning model and world models in terms of sample efficiency, obtained rewards, and the number of steps to finish a task. I believe this work studies a promising research direction (i.e. program-guided RL) and proposes an interesting framework to address the task. I would like to see this paper included in the conference while also hope the authors address my questions, including providing more comprehensive analyses are required to justify some design choices and adding missing relevant works.
main_review: ## Paper strengths and contributions
**Motivation**
- The motivation for solving a long-horizon task in a partially observable environment by first synthesizing a high-level plan is convincing.
- The idea of utilizing programs as a representation to describe high-level plans that need to be executed to fulfill a given task specification is intuitive. This paper presents an effective way to implement this idea.
- The limitations of prior works are explicitly mentioned, which strengthens the motivation of this work: (1) most of symbolic planning works cannot deal with continuous state and action spaces or partially observable environments, and (2) deep RL methods struggle at learning long-horizon tasks.

**Technical contributions**
- Hallucinating observable parts of the environment by predicting abstract state variables instead of concrete states seems effective.
- Formulating the problem of fulfilling a given task specification as a MaxSAT problem and utilizing a MaxSAT solver to synthesize the plan is convincing.

**Clarity**
- The overall writing is very clear. The organization is easy to follow. The authors utilize figures well to illustrate the ideas.
- Motivating Example Section not only provides a clear description of the family of problems considered in this paper but also presents the intuitions of the proposed framework. This writing flow makes the paper easier to read.

**Related work**
The authors give a clear description of the related prior works from both the perspectives of reinforcement learning and planning.

**Ablation study**
Ablation studies are helpful for understanding some design choices and the effectiveness of submodules. 
- Table 6 verifies the advantage of using a learned hallucinator.
- Figure 7 justifies the proposed CVAE which allows for sampling multiple possible outcomes, while the margins are small.

**Experimental results**
- The experimental results demonstrate that the proposed framework outperforms a DRL baseline and a world model baseline and performs similarly to an oracle method (program guided agent) which utilizes a given ground truth program.
- The authors, instead of evaluating on a single domain like most of the related works, evaluated the proposed framework and the baselines in two domains (I consider the 2D-craft environment and Ant-craft environment the same), making the results more convincing. 

**Reproducibility**
The code is provided, which is helpful for understanding the details of the proposed framework.

## Paper weaknesses & questions

**The performance breakdown**
Only the overall performance of the proposed framework is shown in the paper. It is not clear how well each module (i.e. the hallucinator and the executor) performs. It would be insightful to break down the overall performance.
- For the hallucinator, I would like to see how accurate (in %) it can predict the unobserved parts of the environments so that I would know better how much error comes from the prediction. With this measurement, it would be easier to justify the design choice of the hallucinator (see below).
- For the executor, I would like to see if the option policy can always complete a given component so that the failure can be attributed to either the predictions of the hallucinator or the execution.

**Design choices of the hallucinator**
Only the comparisons of the proposed hallucinator, an optimistic hallucinator, and a random hallucinator are provided in Table 6. This only justifies the necessity of learning the hallucinator. However, it is not clear if training a CVAE for this task would yield the best results. I would like to see the performance of the following:
- A deterministic baseline: instead of predicting a distribution allowing sampling, it would make sense to compare against a naive auto-encoder model that produces deterministic predictions so that we can verify the intuition of using CAVE to handle multiple possible outcomes.
- VQ-VAE: I am wondering if using a conditional VQ-VAE would yield better results especially given the state features are discrete which aligns well with VQ-VAE discrete latent codes.
I understand the tight timeline of the rebuttal so I am not demanding the abovementioned results. I simply believe including it would make the paper stronger and more convincing.

**Predict concrete states**
While it is intuitive to predict abstract feature states in this case, it would be informative to justify this choice (i.e. train a hallucinator that predicts raw states).

**Design choices of the executor**
The authors proposed to use modular networks for the executor. I wonder if it actually yields better results compared to a simple multitask policy trained to solve all options. Again, I am not asking the results for the rebuttal.

*Executor training*
Why not training the executor by simply sampling a subgoal from all the possible subgoals instead of training it to solve the entire episode, especially given that the reward signal is given whenever a subgoal is completed? 

**Missing related work: program synthesis**
The related work covers two lines of works: reinforcement learning and planning. However, I feel an important line of prior works is missing: program synthesis. Program synthesis methods aim to automatically synthesize programs from high-level task specifications. The program synthesis problems considered in this work can be cast as solving MaxSAT problems with a solver. Yet, this is not always the case and this is also when program synthesis works play a role - to search a program or learn to synthesize a program. Therefore, I believe it would be important to discuss this line of work. Some example papers are as follows:
- RobustFill: Neural Program Learning under Noisy I/O
- Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis
- Synthetic Datasets for Neural Program Synthesis
- Learning to Describe Scenes with Programs
- Neural Program Synthesis from Diverse Demonstration Videos
- Learning to Infer and Execute 3D Shape Programs
- Improving Neural Program Synthesis with Inferred Execution Traces
- Execution-Guided Neural Program Synthesis

**MaxSAT assumption**
This paper assumes the program synthesis problem can be cast as a MaxSAT problem, which is not always the case. I would like to know what authors think about how the proposed framework can be modified to solve problems that do not fulfill this assumption. 

**Ant-craft**
Since the low-level controller of the ant is pre-trained. I am not sure what the difference between the Minecraft environment and the ant craft environment. But, maybe it is helpful for the readers who do not understand the intuition (i.e. decouple the symbolic part and the neural network policies) which allows the proposed framework to deal with continuous state and action space

**Code**
When I tried to reproduce the results, there are some missing dependencies that are not mentioned in the README, including pyyaml, matplotlib, tqdm. It would be easier to provide a requirement text file to include those.

**Minor**
Figure 6: should be Table 6

## Other metrics

### Relevance and significance
- Solid contribution to a relevant problem

### Novelty
- One idea that surprised me by its originality, solid contributions otherwise

### Technical quality
- Technically adequate for its area, solid results

### Experimental evaluation
- Sufficient evaluation w.r.t. most criteria

### Clarity
- Very clear, only minor flaws
limitations_and_societal_impact: Described in the main review section.
review_title: null

Meta-review: All reviewers agree that the paper tackles an interesting research problem, ie long-term planning in POMDPs. However, multiple issues were raised and remain after the discussion, among these are clarity of exposition and mostly importantly novelty of contribution. Concerning the latter, introduction of a generative model into the architecture for modeling belief states in partially observed environments has been addressed more thoroughly in the literature before (eg see [1] and references therein) mostly without the need for privileged information during training. The problem of planning in POMDPs has an even richer history, eg the method proposed in [2] would be directly applicable to the setting considered here. However the method is not referenced nor compared to in experimental results. The paper is therefore not accepted in its current form.

[1] Temporal Difference Auto-Encoder. Gregor at al. 
[2] Monte-Carlo Planning in Large POMDPs. Silver, Veness.
Decision: Accept (Spotlight)