Program Synthesis Guided Reinforcement Learning for Partially Observed Environments

Abstract: A key challenge for reinforcement learning is solving long-horizon planning problems. Recent work has leveraged programs to guide reinforcement learning in these settings. However, these approaches impose a high manual burden on the user since they must provide a guiding program for every new task. Partially observed environments further complicate the programming task because the program must implement a strategy that correctly, and ideally optimally, handles every possible configuration of the hidden regions of the environment. We propose a new approach, model predictive program synthesis (MPPS), that uses program synthesis to automatically generate the guiding programs. It trains a generative model to predict the unobserved portions of the world, and then synthesizes a program based on samples from this model in a way that is robust to its uncertainty. In our experiments, we show that our approach significantly outperforms non-program-guided approaches on a set of challenging benchmarks, including a 2D Minecraft-inspired environment where the agent must complete a complex sequence of subtasks to achieve its goal, and achieves a similar performance as using handcrafted programs to guide the agent. Our results demonstrate that our approach can obtain the benefits of program-guided reinforcement learning without requiring the user to provide a new guiding program for every new task.


Reviews and author responses:

This paper builds upon previous work using a guiding program over options trained with Reinforcement Learning to solve new tasks in an environment that requires planning.
It adds capabilities to deal with partially observable environments and reduces the burden in the user by introducing a world model (hallucinator) and a program synthesiser.
In the environments used by the authors, the new method, model predictive program synthesis (MPPS), perform on par with a method that has access to the ground truth program (i.e. requiring extra user intervention) and outperforms baselines that don't use programs.
Furthermore, the authors demonstrate the capacity of the learned policies and programs to be transferred to a continuous actions domain by replacing the grid world agent with a MuJoCo ant.  I appreciate this direction of work that brings programs and DNN together to solve new tasks and increase the planning capabilities of current agents. I am particularly excited about the use of MaxSAT solver to optimise over possible programs to solve the current goal.
The extensions brought by this work are useful and in the right direction, but I am not satisfied that they are enough to warranty an acceptance to this venue.

Training world models at the right level of abstraction is still a fairly difficult task and this work uses simplistic setting by having access to the ground truth state of the world the train the generative model. Building the correct state abstraction from observations is crucial in order to plan well. It's unclear how the proposed approach would cope without access to the state privileged information.

Another unsatisfactory aspect of the approach to handle partially observable environments is the exploration aspect of the policy. The authors assume that by executing one of programs initially proposed, the agent will explore and gather more information about the environment, but this seems to be an artefact of the environments used by the authors. In these environments any policy, even a handcraft exploration policy, will yield good exploration behaviour, gathering useful information to improve future plans. This is not generally true. In certain environments, the optimal way to gather information is by actively exploring the environment and striking the balance between exploration and exploitation is crucial to achieve good performance and data efficiency.
One approach is to exploit the uncertainty of the world model to drive exploration, such as in "Plan To Explore".

Again, I am excited about the direction that this work is taking and I look forward to see further improvements and applications in more challenging scenarios. I appreciate that the authors described related work in good breadth, but my understanding is that some of these other lines of work address different problems to the one that the authors are trying to solve, for example in terms of access to privileged information.
I would suggest to the authors to introduce a "Limitations" session to their work addressing some of the points discussed in my main review. They could then position themselves better with respect to related work in terms of advantages and disadvantages of their work.

 Thanks for the rebuttal; a good part of my questions has been answered while larger observation spaces and distribution mismatch for the skills remain for future work. I will remain with a positive score (which is close to an upgrade but sadly not sufficient given limitations above) and argue for acceptance of the paper.

 We performed an additional experiment on a non-deterministic version of 2D-craft. Here, each action has a 20% chance of failing (when a move action fails, the agent move to a random direction; when a use action fails, the action becomes a no-op). The performance on the test set after training (400k episodes) is:

| Baseline | Avg. reward | Avg. finish step |
| ------ | ------ | ------- |
| End-to-end | 0.22 (0.02) | 83.3 (1.8) |
| World models | 0.20 (0.01) | 83.6 (0.7) |
| Ours | 0.47 (0.03) | 73.1 (1.2) |
| Oracle | 0.50 (0.03) | 69.9 (1.6) |

We can see that all the approaches take longer time to solve the tasks when the environment becomes non-deterministic. This is because the environment becomes more challenging. Our approach still significantly outperforms non-program-guided baselines, and performs comparably with the oracle baseline. Note that for the non-program guided baselines, the average reward (ratio of test tasks successfully solved) does not change significantly, since they fail to solve the challenging tasks even when the environment is deterministic.

We will add this study to our paper. Thank you for the suggestions! Please let us know if you have any further questions!

 > In the meanwhile, I have finished reading all other reviewer's comments and your responses. One commonality in the responses to all 4 reviewers is the mention of the multi-task/train-then-test scenario. I went through the paper again and the "test set" is only introduced in 6.3 at the end of the paragraphs for 2D-craft and Box-world. I think the paper would be clearer if this explicit held-out set setup is more clearly denoted either in the problem formulation, or probably more fittingly, denoted in a header section for 6.3/6.1/6. Details about how the training and test sets are created (even if purely random) should be put into an appendix or in this header section.

> This would clear up some confusion and improve the paper, as this detail was easy to miss/forget as all 4 of us reviewers might've.

This is a great suggestion, thank you! We will add a dedicated section to describe this multi-task setup. Thanks a lot for these valuable suggestions! 

We will update the results here after it finishes running.

 Thanks for the response. I will happily await the new results.

In the meanwhile, I have finished reading all other reviewer's comments and your responses. One commonality in the responses to *all 4 reviewers* is the mention of the multi-task/train-then-test scenario. I went through the paper again and the "test set" is only introduced in 6.3 at the end of the paragraphs for 2D-craft and Box-world. I think the paper would be clearer if this explicit held-out set setup is more clearly denoted either in the problem formulation, or probably more fittingly, denoted in a header section for 6.3/6.1/6. Details about how the training and test sets are created (even if purely random) should be put into an appendix or in this header section.

This would clear up some confusion and improve the paper, as this detail was easy to miss/forget as all 4 of us reviewers might've.

 We thank the reviewer for the reply and clarifications!

> I don't believe that this should be analyzed in future work. It seems to be very fitting to be analyzed in this work as this MaxSAT solver is a key component of this method, for example by making the gridworld environment nondeterministic.

We agree with the reviewer, and we will perform an additional experiment on a non-deterministic version of the grid world environment. We will update the results here.

> I agree with this specific statement the authors make. However, there may be a misunderstanding based on what i stated in the original review, which I apologize for if it wasn't clear. The line I referenced in my original review comes right after "However, these approaches cannot be applied to MDPs with continuous state and action spaces or to partially observed MDPs." The works I referenced demonstrate that option-planning works are able to be applied to continuous state spaces, making the first part of this statement false. Furthermore, unless the authors demonstrate that they "cannot be applied" to POMDPs, then the second part of the statement is unproven. I highly recommend that the authors remove or tone down this statement and/or include some of works I referenced.

We agree with the reviewer here. We will tone down this statement and include the works that the reviewer referenced in our paper. Thank you for the clarifications and suggestions!

 I thank the authors for their detailed reply. My responses to some of the points follows:

> Our MaxSAT synthesis works with the logical formulae of the components that describe their intended behaviors. These logical formulae can be used even if the underlying environment is non-deterministic. Non-deterministic environments might, however, increase the difficulty of the executor, since it must still learn to achieve the desired behavior in the face of a non-deterministic environment. We believe that studying how the executor performs in non-deterministic environments and potential ways of improving the learning under such cases is an interesting direction for future work. We thank the reviewer for the suggestions.

I don't believe that this should be analyzed in future work. It seems to be very fitting to be analyzed in *this* work as this MaxSAT solver is a key component of this method, for example by making the gridworld environment nondeterministic. 

> At a high-level, many of these papers do not handle partial observability or partially observable environments are not the main focus. Natural ways to leverage these approaches in partially observed environments is to either ignore the partial observability, or to plan in an imagined future.

I agree with this specific statement the authors make. However, there may be a misunderstanding based on what i stated in the original review, which I apologize for if it wasn't clear. The line I referenced in my original review comes right after "However, these approaches cannot be applied to MDPs with continuous state and action spaces or to partially observed MDPs." The works I referenced demonstrate that option-planning works are able to be applied to continuous state spaces, making the first part of this statement false. Furthermore, unless the authors demonstrate that they "cannot be applied" to POMDPs, then the second part of the statement is unproven. I highly recommend that the authors remove or tone down this statement and/or include some of works I referenced.

Other than these points, thank you for clarifying the other questions I had in my review.

 We thank the reviewer for their thoughtful and insightful review! We are excited to see your appreciation on our motivation, novelty, and technical contributions. We also appreciate the constructive suggestions by the reviewer, which point to some interesting research directions as well as how to further improve the paper.

We answer the reviewer's questions in the following.

### Transfer experiments

We agree with the reviewer that studying the transfer to significantly different tasks from the training tasks is an interesting direction for research. We want to point out that we do use a held-out test set for all the evaluations (training curves, Table 1, etc.) in our experiments; in particular, this test set consists of (goal, world) pairs that are unseen during training. We thank the reviewer for this insightful suggestion on an interesting research direction.

### Deterministic environments

Our MaxSAT synthesis works with the logical formulae of the components that describe their *intended* behaviors. These logical formulae can be used even if the underlying environment is non-deterministic. Non-deterministic environments might, however, increase the difficulty of the executor, since it must still learn to achieve the desired behavior in the face of a non-deterministic environment. We believe that studying how the executor performs in non-deterministic environments and potential ways of improving the learning under such cases is an interesting direction for future work. We thank the reviewer for the suggestions.

### Ablations

We agree with the reviewer that studying whether and how to make the hallucinator directly predict the entire world is an interesting and meaningful direction. Compared with state features, the concrete world state is more unstructured and potentially sparse, which might cast new challenges to the training of the conditional generative model. We also want to point out that many existing works in planning and RL also assume that the state abstraction is given, e.g.:
- Hierarchical Task and Motion Planning in the Now, Leslie P. Kaelbling et.al., ICRA 2011
- Value Preserving State-Action Abstractions, David Abel et.al., AISTATS 2020

And there is a long history of work on automatically learning/generating state abstractions for planning, e.g.:
- Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining, George Konidaris, Andrew Barto, NIPS 2009
- Automatically Generating Abstractions for Planning, Craig A. Knoblock, Artificial Intelligence, 1994

Therefore, utilizing state abstractions has been recognized as an effective approach for planning and RL. And the advancement in the above-mentioned area can further facilitate approaches using state abstractions to be applicable to a broader set of tasks and scenarios.

### Related works

Thank you for sharing these papers from the options literature. At a high-level, many of these papers do not handle partial observability or partially observable environments are not the main focus. Natural ways to leverage these approaches in partially observed environments is to either ignore the partial observability, or to plan in an imagined future. Ignoring partial observability in our benchmarks does not work, since there is no valid plan. As we show, planning in a single imagined future is not robust. Our MPPS approach based on MaxSAT synthesis provides a way to synthesize guiding programs that robustly account for multiple possible futures. We will add an expanded discussion to our paper.

Generally speaking, hierarchical RL with continuous state/actions and partially observed settings remains a challenging yet important problem, and our paper proposes a novel approach for solving it by combining conditional generative models and program synthesis techniques.

### Minor issues

We thank the reviewer for the careful reading and kind suggestions. We will edit our paper accordingly.

### Observations

The observations we give to the learned neural policies $\pi$ and the CVAE model are the map with all the previously observed grid cells up to the current step. The example figures in Figure 4 of the paper show what this is referred to (without the green arrows). Each grid cell in the map is a one-hot vector representing which item is in that cell. The unobserved parts of the map are represented by a special entry in the one-hot vector. We thank the reviewer for the clarification questions. We will further clarify this point in the paper.

Again, we thank the reviewer for the insightful suggestions! We hope that we have addressed the reviewer's questions. We are happy to answer any further questions!


 We thank the reviewer for the comments. We are glad you find our proposed overall architecture quite interesting. We present here the answers to your questions and further experimental results as requested. We hope that this will address your questions and concerns.

### Partial Observability

As suggested by the reviewer, we performed an additional experiment in 2D-craft where the agent first randomly explores the map until the zone it is currently within is all observed, then it synthesizes the program and follows it. The results are shown as follows:

| Approach | Average Reward | Standard Error |
| --------- | --------------- | --------------- |
| Random explore | 0.024 | 0.004 |
| Ours | 0.70 | 0.03 |

This shows that random exploration then planning is not effective enough for our tasks -- our tasks do require the policy to do exploration effectively and synthesizing guiding programs under partial observations.

We would also like to point out that many existing approaches on partially observed environments also only experiment on symbolic, grid-world environments [1, 2, 3]. Moreover, the 2D-craft environment we consider is already a harder version than what is used in previous works: [4] uses 8x8 maps, fully observable, and no zone structure; [5] uses 8x8 maps, fully observable, with zone structures; our work uses 10x10 maps, partially observable, with zone structures. As we show in our experiments, existing reinforcement learning algorithms without program guidance are unable to solve our tasks effectively; harder tasks would only widen the performance gap.

### State Features

The transition from R^n to the set of logical variables is performed using an user-provided abstraction function. For example, for the minecraft benchmark, this abstraction function first identifies regions in the map that are separated by stones or water and then counts the resources and workshops in each region. The set of logical variables are user-defined as well. Our work assumes that these abstractions are given. The hallucinator is trained to predict these abstracted states directly, since our synthesizer works with these abstracted states.

We want to point out that many existing works in planning and RL also assume that the state abstraction is given, e.g. [6, 7]. Automatically learning/generating these state abstractions is an important research area by its own, with a long-history of work, e.g. [8, 9].
The focus of our work is on how to synthesize the guiding programs under partial observation. Therefore, we see the problem of automatically learning state abstractions as an orthogonal and complementary research area to our work.

As for the user burden, writing the state abstraction function only needs to be done once per environment, whereas writing a program needs to be done for every new task (goal and world configuration) in this environment. If we only consider a single task, we agree that writing a program might be simpler. But for the multitask settings we consider, there is a large (potentially infinite) number of tasks, so the once-for-all effort to provide the abstractions is much easier than having to write a program every time we have a new task; this setup is similar to prior work, e.g. [5].

We are happy to include discussions on these topics in our paper.

### Executor

In general, the synthesized program describes a high-level strategy to solve the task, but it does not specify how to achieve it using low-level actions. Therefore, the executor networks are needed to learn how to achieve the guiding programs using low-level actions. For example, in 2D-craft, the executor for “get wood” needs to learn how to explore when it does not see a wood, and which wood to get when it sees multiple of them, etc. Learning neural network policies to execute high-level programs is also used in prior work, e.g. [4, 5].

We use programs synthesized from fully observed maps during training to keep training simple; our approach can be adapted to interweave learning the executor and using program synthesis to plan, but doing so would be slower due to the need to run synthesis during training.

We believe it is reasonable to assume access to fully observed information for the training environments. In particular, we are considering the multi-task setting, where we have a set of sampled training tasks (goal & world configurations) and a held-out set of testing tasks. The goal is to learn a policy using the training set that achieves good performance on the test set. In such settings, it is common practice to use ground truth information of the training set during training, e.g. [5]. This will not invalidate the findings since all the evaluation results (training curves, Table 1 etc.) are evaluated on the test set, where no ground-truth information is given.

Finally, we performed an additional experiment where we trained the baselines (end-to-end, world models) with fully observed maps for 2D-craft. The performance at the end of training (400k episodes) are as follows:

| Approach | Average Reward | Average Finishing Step |
| ------------ | -------------- | ------------------- |
| End-to-end | 0.14 | 87.6 |
| World models | 0.14 | 88.0 |
| Ours | 0.70 | 56.4 |

These results show that the fully observed map information during training does not help the baselines learn faster or better, and is therefore not the reason behind the performance difference between our approach and the baselines.

### Application beyond symbolic worlds

First, as the above-mentioned experiments (random exploration then planning) show, our tasks cannot be transformed into pure planning problems in a trivial manner.

Furthermore, much of the existing work on partially observed environments only experiments on symbolic, grid-world environments, e.g. [1,2,3].

In addition, the 2D-craft environment we consider is already a harder version than what is used in previous works: [4] uses 8x8 maps, fully observable, and no zone structure; [5] uses 8x8 maps, fully observable, with zone structures; our work uses 10x10 maps, partially observable, with zone structures.

Finally, we agree with the reviewer that it would be interesting to study our approach in more complex settings beyond symbolic worlds, as this could potentially bring new challenges and encourage new techniques. We will add a discussion for future work in our paper.

We hope that our answers and additional experimental results have addressed the reviewer's concerns. We are happy to discuss more if the reviewer has further questions.

[1] Learning Reward Machines for Partially Observable Reinforcement Learning, Rodrigo Toro Icarte et.al., NeurIPS 2019

[2] A Reinforcement Learning Algorithm in Partially Observable Environments Using Short-Term Memory, Nobuo Suematsu and Akira Hayashi, NeurIPS 1998

[3] Inverse Reinforcement Learning in Partially Observable Environments, Jaedeug Choi and Kee-Eung Kim, JMLR 2011

[4] Modular multitask reinforcement learning with policy sketches, Jacob Andreas et.al., ICML 2017

[5] Program Guided Agent, Shao-Hua Sun et.al., ICLR 2020

[6] Hierarchical Task and Motion Planning in the Now, Leslie P. Kaelbling et.al., ICRA 2011

[7] Value Preserving State-Action Abstractions, David Abel et.al., AISTATS 2020

[8] Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining, George Konidaris, Andrew Barto, NIPS 2009

[9] Automatically Generating Abstractions for Planning, Craig A. Knoblock, Artificial Intelligence, 1994


 We thank the reviewer for their constructive comments! We are delighted that you find the direction of our work and our MaxSAT synthesis technique exciting. In the following, we provide answers and additional experimental results to address your questions. Hope that will address the concerns you have.

### State abstraction and access to ground-truth state

We agree with the reviewer that having a right level of abstraction is important in order to plan well.  Like much of the existing work in planning and RL, we assume that the state abstraction is given, e.g.:
- Hierarchical Task and Motion Planning in the Now, Leslie P. Kaelbling et.al., ICRA 2011
- Value Preserving State-Action Abstractions, David Abel et.al., AISTATS 2020

The focus of our work is on how to synthesize the guiding programs under partial observation. Furthermore, automatically learning/generating state abstractions for planning is an important research area with a long-history of work, e.g.:
- Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining, George Konidaris, Andrew Barto, NIPS 2009
- Automatically Generating Abstractions for Planning, Craig A. Knoblock, Artificial Intelligence, 1994

We see the problem of automatically learning state abstractions as an orthogonal and complementary research area to our work.

The reviewer also asks about having access to the ground truth state of the world when training the generative model. Our work considers a multi-task setting, where we have a set of sampled training tasks (goal & world configurations) and a held-out set of testing tasks. The goal is to learn a policy using the training set that achieves good performance on the test set. In such settings, it is common practice to use ground truth information of the training set during training -- this is consistent with existing work, e.g.:
- Program Guided Agent, Shao-Hua Sun et.al., ICLR 2020
- Modular multitask reinforcement learning with policy sketches, Jacob Andreas et.al., ICML 2017

This will not invalidate the findings since all the evaluation results (training curves, Table 1 etc.) are evaluated on the test set, where no ground-truth information is given. We believe that for many real-world problems, these assumptions are reasonable since we can exhaustively explore the training environments.

We are happy to add these discussions to our paper.

### Exploration aspect of the policy

Acting optimistically with respect to the predicted environment is known to be a good exploration strategy for reinforcement learning (i.e., “optimism in the face of uncertainty”, including the RMax and UCRL algorithms). Our algorithm can be seen as an instance of the Thompson sampling strategy for exploration, which is also known to be effective for finite MDPs:
- Osband et al., (More) Efficient Reinforcement Learning via Posterior Sampling. NeurIPS 2013.

Furthermore, in our approach, the user has control over what components/options to define. For our current environments, we found the existing options already demonstrating effective exploration. For instance, an option such as “find wood” encourages the agent to explore the environment to achieve its goal. Our executor thus learns to balance exploration and exploitation to achieve this goal as quickly as possible. Generally speaking, for options in partially observed environments, the agent must learn to explore to achieve its goal.

Finally, to demonstrate that our algorithm effectively performs exploration, we conducted an additional experiment in 2D-craft, where the agent first randomly explores the map until the zone it is currently within is all observed, then it synthesizes a program and follows it. The results are shown as follows:

| Approach | Average Reward | Standard Error |
| --------- | --------------- | --------------- |
| Random explore | 0.024 | 0.004 |
| Ours | 0.70 | 0.03 |

We can see that our approach significantly outperforms the random exploration + planning approach. This shows that our tasks do require the policy to do exploration effectively, and our approach demonstrates effective exploration.

### Related work

As suggested by the reviewer, we will add the discussions on the above topics to the paper. We will also further clarify the comparison to the related work as follows.

Much of the existing related literature focuses on the planning setting, where significant assumptions are made about the simulator and access to pre/post conditions for options. The goal is to solve the computationally challenging problem of long-horizon planning on tasks that require complex sequences of subtasks. In comparison to this line of work, our key contributions are providing a general-purpose way to handle partially observed environments (i.e., model predictive program synthesis).

In contrast, most of the recent reinforcement learning literature has focused on short-horizon, low-level control tasks such as walking and running. These approaches do not scale to long-horizon tasks, where additional guiding information is necessary to enable tractable search over plans. We will make the connections to these two literatures more clear in our paper.

We hope that our response addresses the reviewer's concerns. We are happy to discuss more if the reviewer has further questions!


 We thank the reviewer for their insightful and detailed review! We are encouraged by your appreciation of our motivation and contributions. We deeply appreciate your constructive suggestions on how to further improve the paper. We are happy to address your questions and comments as follows:

### Related work

**Connections to planning in partially observable domains:**
We thank the reviewer for pointing out and we are happy to add discussions on this as follows. These approaches assume that the effective state space is small (i.e., number of abstract states), which enables them to compile the problem into a concrete POMDP which can be efficiently solved using POMDP algorithms. However, in general, the effective state space can be exponentially large in the number of predicates (i.e., the abstract state is a bitvector where each bit encodes the truth value of a single predicate). Effectively, we are reducing planning to constraint solving, which is effective since MaxSAT solvers are able to solve problem instances involving very large numbers of binary variables.

**Options framework:**
As far as we can tell, these approaches do not handle partially observed state spaces. Even so, our approach differs in that we are learning options with pre/post conditions, which enable us to perform model-based planning at the high-level. In contrast, these approaches use model-free learning, which can be very inefficient. We are happy to clarify this point in our paper.

### Storing previous observations

The environments we experimented on are static, so the agent only needs to remember the previously observed map regions (they contain all the information of the previously observed states). We agree with the reviewer that in more complex environments the amount of information the agent needs to store can potentially be large. One advantage of using abstract state features for synthesis in our approach is that the agent can store just the abstract state features, which could help alleviate the above-mentioned storage burden. For instance, for a 3D environment, the robot may first run an object detector to determine the available objects and store only this symbolic information. Directly learning over pixels rather than symbolic representations would be an interesting direction for future work; we will add a discussion.

**Side note about section order:**
We thank the reviewer for the suggestion on the structuring of the paper. We will do our best to incorporate their suggestions into our paper.

### Generating options upfront

We agree with the reviewer that in certain settings one might need to train the options in a continual manner to prevent distribution shift. Our approach can be used to interweave option learning and program synthesis, but we found that we did not need to do so for our benchmarks. Studying our approach in a continual/online learning setting is an interesting future research direction. We will add this discussion in our paper.

### Further questions on results

**Training Curve:**
We thank the reviewer for pointing out, and we will add the training episodes for the hallucinator to the training curves. Note that the number of training episodes for the hallucinator are much lower than the number of training episodes for the executor:

| Environment | #Episodes for hallucinator | #Episodes for executor | hallucinator/executor (%) |
| ---------------- | ------------------ | ------------------- | ------------ |
| 2D-craft | 200 | 400,000 | 0.05% |
| Box-world | 1,000 | 200,000 | 0.5% |

We can see that the number of training episodes for the hallucinator is only 0.05% of the number of training episodes for the executor in 2D-craft, and 0.5% in box-world. So adding the hallucinator training steps to the training curve will not substantially change the training curves.

**Mujoco state abstraction:**
MuJoCo’s state abstraction is similar to the 2D-craft state abstraction. In our experiments we train the hallucinator to directly predict the abstract states, since our synthesizer works with abstract states. We believe that in many applications, high-level states can be represented in a symbolic form (e.g., by running an object detector to determine available objects and their attributes); continuous control is most useful for learning low-level skills that are hard to represent symbolically. Finally, we note that the hallucinator can in principle operate over continuous spaces -- it is a conditional generative model, and there is a large body of work on learning conditional generative models over continuous spaces. We thank the reviewer for the clarification question and we will add this discussion in our paper.

**Relational baseline only on boxworld:**
This is because baseline [38] was originally proposed for boxworld. Their paper [38] considers the fully observed version of boxworld, where we experimented on the partially observed version.

**Other program synthesis baselines:**
Our MaxSAT program synthesis method is able to accurately solve the problems with guarantees (solving a program that satisfies a maximum number of hallucinated worlds). Works on program synthesis typically design efficient synthesis algorithms specialized to a particular problem/DSL, e.g. the famous FlashFill paper:

- Automating String Processing in Spreadsheets Using Input-Output Examples, Sumit Gulwani, POPL 2011

Alternative algorithms such as enumerative search are not scalable to our search space: for example 2D-craft contains 1.7 million programs up to length 6, and this search space grows exponentially as program length increases. We are not aware of an effective alternative synthesis approach that is suitable as a baseline for the problems we have.

**Relation between [2] and [31]:**
First, [2] works on fully observed environments, and needs user-provided straight-line programs for every (goal, world) pair. They do not assume that the program semantics are known. Next, [31] follows up on [2] by proposing to incorporate the program semantics into learning, and show that this strategy can substantially speed up learning.
We use a variant of [31] as the oracle since it learns much faster than [2] (due to its stronger assumptions); the only modification is that rather than give it the full program with conditionals, we give it the straight-line program specialized to the current task/environment.

**Expand results section & why craft environment hard for non-program guided baselines:**
We thank the reviewer for the suggestions and will expand the results discussion. The craft environment has a long horizon, sparse rewards, and involves compositional strategies that require logical reasoning to obtain. Hence, it is very hard for all non-program guided baselines, which is consistent with empirical results in prior work [2, 31].

### Minor suggestions for improvements
We thank the reviewer a lot for these helpful and constructive suggestions! We will incorporate them into our paper. Thank you!

We hope the above response addresses your questions. We are happy to answer any additional questions you might have!


The submission proposes a method for program synthesis in partially observable domains in reinforcement learning. The approach aims to handle uncertainty regarding the environment by using three key components: a hallucinator module, a synthesiser module and an executor module.

The hallucinator is a conditional generative model aiming to predict likely states of the environment. These are used by the synthesiser to generate a program which finally is used by the executor which runs the corresponding subroutines in the order given by the program.

The method is evaluated on three benchmarks including 2 gridworlds with discrete action space and 1 MuJoCo environment with continuous action space. Across these domains, the proposed method performs better than a set of non-program synthesis baselines. The experiments finish with a set of ablations for the hallucinator.
  The paper is overall well written, clear and easy to follow. The structure is understandable and the problem is well defined. The method performs better than a set of baselines and the additional ablations are quite helpful in understanding the key component of the paper: the hallucinator which enables addressing partially observed environments.

For the related work, I have to acknowledge that I am less familiar with STRIPS planning in POMDPs but connections to older work relating planning in partially observable domains or in connection with uncertainty such as [1, 2] would be helpful.
In addition, the statement about the options framework is not accurate as it has been adapted in many forms to also address partial observability and continuous actions [3,4].

Regarding the experiments, there are a couple of open questions, in particular on the handling of partial observability. Why does the environment automatically store all previously observed states (L99)? This might work into simple 2d environments but would completely break in 3D environments where significant amounts of data would be generated. On a side note, it might make sense to first complete the problem definition before giving an example which uses the formalisms which are only later introduced (Section 2 could be better placed after Section or even later).

For a more general (and continuous/non-simplified domain) the generation of all options in advance might be challenging or impossible as policies need to be trained for all possible initial conditions in order to prevent distribution shift (and problems with generalisation). Even if skipped in the main paper, it would be relevant to mention this as a shortcoming and point towards future possibilities.

Further questions focus on the results:
Are the training episodes for the hallucinator (random agent data) included in the plots? if not then they should be added as this is data not available to other agents.
Is the state in MuJoCo as abstracted as in the other domains such that the hallucinator does not actually handle continuous spaces? This would be a strong limitation as well.
Why is baseline [38] only applied to boxworld?
Why no other program synthesis methods were used as baselines in these sections?
How do [2] and [31] relate? The question comes up since [2] was already applied to the craftworld domain while [31] is chosen as the oracle baseline here.
Only a single paragraph is spent on the main results (beginning 6.4). It would make sense to extend this and elaborate on further details such as why the craft environment seems to be very hard for all non-Oracle baselines. (see later suggestion for freeing space to include these comments.)

(With the assumption of minor corrections) the paper is clear in its contributions, addressed a relevant point and could be a relevant addition to this year’s conference.

Minor suggestions to improve accessibility of the paper
- Define ‘option’ before using it in line 27.
- Figure 1 could be moved out of the introduction and into the method section 
- L162 maybe -> may be
- This is a question of taste but sections 4 and 5 iterate over the same model components and it might be useful to merge the text about learning directly into 4 as well.
- The implementation details 6.3 could be moved to the appendix to make space for more important points.


[1] Bonet, B. (1998). High-Level Planning and Control with Incomplete Information Using POMDP's.
[2] Draper, D., Hanks, S., & Weld, D. S. (1994, June). Probabilistic planning with information gathering and contingent execution. In AIPS (pp. 31-36).
[3] Zhang, S., & Whiteson, S. (2019). DAC: The double actor-critic architecture for learning options. arXiv preprint arXiv:1904.12691.
[4] Wulfmeier, M., Rao, D., Hafner, R., Lampe, T., Abdolmaleki, A., Hertweck, T., ... & Riedmiller, M. (2021, July). Data-efficient hindsight off-policy option learning. In International Conference on Machine Learning (pp. 11340-11350). PMLR.
 The paper adequately mentions limitations.

The paper proposes a framework for program synthesis guided RL that is applicable to partially observable environments. There 3 core components of the proposed approach:
 i) halluncinator - a generative world model approximating p(state | observation) which is pre-trained on the world; 
ii) synthesizer - a program synthesizer based on a MaxSAT solver that can generate programs reaching a desired goal from hallucinated states;
iii) executor - a modular neural network that is trained to execute the synthesized programs.
The authors demonstrate the capabilities of the proposed framework on 3 problems (2D-craft, box-world, ant-craft) and report better learning performance then other well established methods.  While the paper does introduce an interesting approach for improving the capabilities of RL agents with program guided synthesis, there are several major issues that I'd like to address.

*Partial Observability.* The authors mention several times that the proposed approach is well suited for partially observable environments and that classic planning methods fail particularly because of the uncertainty in the environment. My biggest concern is that since observations are aggregated and the environment is static it takes very little exploration to turn an initially partially observable environment into a fully observable and completely deterministic environment. For example, one of the considered worlds is 10x10 , the maximum episode time is 100 steps and the agent has 5x5 field of view. In theory, with as little as 10 + 10 + 5 = 25 steps the problem can be turned into perfectly suitable for classical planning. One of the baselines that I feel is missing is an agent that first explores randomly until the full map (or the zone it currently is within) is fully observable and than uses any sort of classical planning. 

*State Features.* The considered POMDP model has states its states in R^n, but many of the explanations involve state features \alpha(s). How is the translation from R^n to a set of logical variables performed? How is the set of these logical variables defined? Does he training of the hallucinator module take into account these logical predicates somehow? If so, how? In many domains beyond grid worlds even just defining these state features requires much more work than writing a short program using these features to solve the problem.

*Executor.* I couldn't really understand what is the role of the executor network. Why is a network needed to execute a policy that follows a synthesized program anyway? Moreover, the Executor section of  5. Learning Algorithm suggests that the executor is pre-trained on trajectories sampled from the program determined from the fully observable environment. This seems to me as a very likely explanation of why the obtained results are so closed to the oracle baseline.  I find the proposed overall architecture quite interesting, but would really like it to be applied to a task beyond symbolic worlds that with a limited amount of effort can be transformed into pure planning problems.



The authors propose to reduce the dependency on user-provided programs for program-guided agents by instead requiring users to submit a goal specification and then automatically synthesizing programs to guide an agent to achieve the given goal specification. The method is novel and a strong technical contribution and the motivation is clear and important. I have some issues with the ablations and experiments that I would like to see addressed in the rebuttal, but I believe this paper should be accepted regardless.  ## Paper Strengths
**Paper Framing/Motivation**
The novelty here is explained well: reducing the supervision/user burden required to create program-guided agents. This is an important motivation, as Program-Guided agents have been shown to be good at performing the types of tasks for the environments tested, but reducing the user burden required to get such approaches to work is important to be able to extend these types of agents to more difficult and realistic environments.

**Technical Contributions**
The paper combines generative world models, program synthesis, and reinforcement learning in a cohesive, sensible, and novel manner to tackle the problem setting of automatically synthesizing programs that can guide (option-conditioned) agents to solve tasks. Furthermore, defining the program synthesis task through a MAX-SAT problem reduction and using dedicated MAX-SAT solvers is intuitive, interesting, and sensible.

**Paper Clarity**
The paper is well-written and notations are generally clear given the complexity of the method. Introduction is very well-written, and intuitions about what the purpose of each component is and how they’re learned are clear. The paper figures do a good job of describing the method and demonstrating visual examples.

**Experiments**
Having 2 different types of environments is great, as a lot of program synthesis works focus on just 1. 

The study of how the number of samples produced by the hallucinator affects the mean and variance of final performance numbers is interesting. It also perhaps signals that a non-sampled hallucinator would work well too. I wonder how many more samples would be needed in non-deterministic environments.

**Results**
The method achieves solid performance gains over baselines and manages to perform closely to the oracle on the evaluated tasks.

**Reproducibility**
Included code is comprehensive and comes with a script to reproduce nearly all experiments and plots.

## Paper Weaknesses
**Experiments**
I think that the Ant experiment is not that meaningful, especially because the main change is the option policies which are instead trained with Soft-Actor Critic. Once trained with SAC, the policies will do the job of guiding the agent through the environment, therefore negating the difficulty of including the Ant with respect to the interesting parts of the method, namely the hallucinator and program synthesizer. I think a more meaningful transfer experiment would’ve been to transfer the agent to a set of 2D craft environments that are completely unseen during training. In this experiment, the authors would be able to demonstrate the ability of: 1) the hallucinator model to generalize to unseen conditions, and 2) the ability of the program synthesizer guide an agent to solve unseen conditions effectively through inductive biases induced by the design of the hallucinator and programs (compared against naive RL).

**Determinstic environments**
I believe in non-deterministic environments, the MAX-SAT problem would be more difficult to solve due to it being a probabilistic, constrained optimization problem. It would be great if the authors had a non-deterministic version of one of the environments to see how the performance changes, as non-determinism in the program execution seems to be a hurdle to getting these types of program synthesis works to work on more realistic or more difficult environments.

**Ablations**
There should be an ablation where the hallucinator predicts the entire world, not just the state features. This would reveal insights into whether the architecture of the hallucinator is sufficiently powerful to extend this work to more difficult tasks in which these abstract state features are not readily available.

**Related Works**
Regarding hierarchical RL works: “Recent work has addressed the former [1, 18, 11] by combining high-level planning with reinforcement learning to handle low-level control, but not the latter…” Unless I’m misunderstanding what the authors are stating, this is not true; recent hierarchical RL works can plan with options/skills in partially observed settings by discovering these skills/options automatically through self-supervised objectives or reward-based objectives (even if they don’t directly apply them to POMDPs or call their skills options, they should still work fine). See for example: Hierarchical Reinforcement Learning by Discovering Intrinsic Options (Zhang et al, 2021), Option Discovery using Deep Skill Chaining (Bagaria and Konidaris, 2020), A Deep Hierarchical Approach to Lifelong Learning in Minecraft (Tessler et al, 2016), Sub-policy Adaptation for Hierarchical Reinforcement Learning (Li et al, 2020).

**Minor Issues**
L107: $s^{-}$ -> $s^{+}$

A couple grammatical issues throughout.

**Questions**
What are the observations given to the learned neural policies $\pi$? In some environments states become more and more observed as time passes, how are these expanding observations represented and given to $\pi$ and the CVAE model?
 Yes.

Meta-review: This is an easy meta-review to write, as the during discussion, the consensus clearly shifted towards a mean of 7 amongst those reviewers who actively responded, with the outlier not having replied to the authors rebuttal. This paper is pretty cool — it proposes a method for program-synthesis guided RL, but does not depend on privileged information in the form of ground truth programs. It performs better than comparable approaches that do not have the inductive bias of program-synthesis within their agents. I'm of the opinion we should see more work in this area at NeurIPS, and the quality of this work definitely warrants it being an exemplar thereof. I recommend acceptance.

Decision: accept