Learning to Synthesize Programs as Interpretable and Generalizable Policies

Abstract: Recently, deep reinforcement learning (DRL) methods have achieved impressive performance on tasks in a variety of domains. However, neural network policies produced with DRL methods are not human-interpretable and often have difficulty generalizing to novel scenarios. To address these issues, prior works explore learning programmatic policies that are more interpretable and structured for generalization. Yet, these works either employ limited policy representations (e.g. decision trees, state machines, or predefined program templates) or require stronger supervision (e.g. input/output state pairs or expert demonstrations). We present a framework that instead learns to synthesize a program, which details the procedure to solve a task in a flexible and expressive manner, solely from reward signals. To alleviate the difficulty of learning to compose programs to induce the desired agent behavior from scratch, we propose to first learn a program embedding space that continuously parameterizes diverse behaviors in an unsupervised manner and then search over the learned program embedding space to yield a program that maximizes the return for a given task. Experimental results demonstrate that the proposed framework not only learns to reliably synthesize task-solving programs but also outperforms DRL and program synthesis baselines while producing interpretable and more generalizable policies. We also justify the necessity of the proposed two-stage learning scheme as well as analyze various methods for learning the program embedding. Website at https://clvrai.com/leaps.


Reviews and author responses:

 We thank the reviewer for the thorough and encouraging review that points out the unclear parts of our submission. Also, we deeply appreciate the reviewer for not only spotting the errors in our DSL but also clearly stating how they should be fixed.

 Many thanks for the clarifications. I'm glad the review was useful

 We appreciate your detailed feedback and the time you spent on helping us to improve this work!

 Thanks a lot for the detailed answers! After going through them carefully I updated my rating to better reflect the quality of the work.

The paper approaches reinforcement learning from a program synthesis perspective and proposes a new method - LEAPS - that is capable of learning programmatic policies just from weak reward signals. This is achieved in two steps:
i) learning a smooth latent manifold where similar programs are encoded close to each other;
ii) searching the latent space efficiently via the Cross Entropy Method.
LEAPS is applied to several tasks in the Karel domain where it successfully can learn programmatic policies that are even transferable to larger state spaces for zero-shot learning.  I enjoyed reading the manuscript and found some of the presented ideas quite interesting. As noted by the authors, the problem of synthesizing programs directly from weak reward signals is quite challenging and so other works utilize entire traces for supervision or resort to program templates. In contrast, the proposed approach is capable of synthesizing programs from the ground up using a predefined DSL that contains loops, conditional statements and domain dependent actions and perceptions.

At the core of this novel capability is the set of behavioural losses used to train a varioational autoencoder resulting in a latent space where programs with similar behaviours are grouped closer to each other. While the proposed program behaviour loss is an obvious step towards building a program autoencoder, the latent behaviour loss is a non-trivial addition which according to the ablation studies makes a sizeable contribution to the performance of the model. How did you choose the architecture of the neural program executor module and were there any speasdcial considerations driven by the DSL and family of supported programs? Once trained how closely did the neural program executor can actually match the execution of the ground truth program?

Another question that I have regarding the behavioural losses is why did you decide to use just actions? Wouldn't action-state sequences provide a stronger supervision signal especially when only fragments of the execution of the reconstructed program semantically match the ground truth trace?

The analyses, experiments and reported results clearly demonstrate that LEAPS does provide RL agents with the capability to learn programatic policies that can even generalize to larger worlds. However, there are a few more results which may be reported that I think would increase the quality of the paper:

1) Traversals of the program latent space and visualizing interpolations between different programs. This will really shed more light on what the learnt latent space looks like.

2) More explicit information about the computational resources and time needed to run the various experiments.

3) At least one plot showing how the best found program evolves over the course of searching - complexity and losses.

Lastly, in figure 15 in the appendix the CEM search is described as converging to the ground truth solution, but the population of candidate solutions in fact passes right by the ground truth and never actually returns to it. Can you explain this phenomena? I think the biggest limitation of the proposed paper is the fact that only relatively short programs are considered. This of course is a result of the simple grid world the Karel domain defines. Therefore, it will be very interesting to see LEAPS applied to tasks that require more complex and compositional behaviours. Additionally, as mentioned by the authors, bringing LEAPS to the robotics domain will be a great step towards synthesizing robust robotics policies, however, I believe for that the LEAPS framework will need to be extended to handle learning perception modules and action controllers simultaneously with the policy search.

 Thank you very much for all your detailed feedback and the time you put into helping us to improve our submission!

This paper suggests representing policies for MDPs as programs, and shows how to learn programmatic policies effectively from a mix of trial-and-error experience on specific tasks and a pre-training phase to create an embedding that maps programs producing similar behaviors into similar points in the embedding. The authors compare their method to a variety of baselines, including both ablations of their own model as well as more traditional approaches from hierarchical RL, deep RL, and a standard program synthesizer on the KAREL domain. The authors find that the programmatic policy representations perform better overall for many of the KAREL tasks, and generalize better to larger problems than they were trained on.  The paper’s pros (thoroughly done scientific experiments, a different perspective on how to build generalizable policies for MDPs, and considered framing and discussion pointing out its own limitations) slightly outweigh its cons (missing related work from classical planning, confusion about the possibility for loops in decision trees, and statistically borderline results in some experiments), and I believe this paper presents a reasonable scientific effort that will be interesting for a subset of the NeurIPS community.

In the next sections I go into pros and cons in more detail, and point to areas that I would like the authors to comment on/address in their revisions. ++ is pro, -- is con.

Related work:
++ The authors do a reasonable job of situating the work with respect to other efforts to synthesize programmatic policies from deep RL agents, program synthesis approaches, and limitations of finite state machines

-- The authors make statements about decision trees not handling loops several times in the paper, but that is a somewhat misleading statement. It is possible to include loops in decision trees, but these loops must be encapsulated differently within the DSL (see Silver et al, AAAI 2020 for an example where they use a “scanning” function that operates like a while loop within a decision tree context). In the author’s DSL, I think this would amount to having a separate “Loop l := while(b, s)” and to make the program “p:= DEF run() : s | l”. There are certainly limitations to this, and these loops cannot be generally as expressive as doing full program synthesis, but I think this statement should be watered down (or removed) within the paper. It might help to reference Silver 2020 to point out the limitations of such setups of while loops in decision trees.

-- There is a missing body of work from classical planning on synthesizing policies as programs in these same structures, two references below
1. For synthesizing for loop and while loop policies: LoopDistill (Elly Winner and Manuela Veloso, 2007)
2. For learning policies as programs in a more expressive space: Learning Teleoreactive Logic Programs from Problem Solving (Choi and Langley)

Results:
++ I appreciated the thoroughness of ablations covering all main aspects of the program embedding learning, and the method was (mostly) explained clearly.
++ There were many comparisons to baselines, and these baselines were mostly appropriately chosen for the domain to underscore the points of the paper (HRL to address extended actions vs. extended actions as programs, DRL-abs to address differences in input representation, VIPER to address at least one alternative program synthesis approach (although there are others that are “neural” and might have been more appropriate, such as REPL, Ellis et al 2019)). The baselines’ hyperparameters were also reasonably optimized, as outlined in the supplement.

++ The authors included exceptionally extensive supplementary material (31 pages!) with some nice visualizations of the learned program embeddings, extra details needed to reproduce the main experiments, extra experiments on transfer to unseen configurations of KAREL tasks, and code.

-- many of the results are statistically insignificant given the enormous standard deviations in the tables, especially in the program reconstruction. While it seems like the full LEAPS model performs statistically better than the naive model, it *does not* seem to perform better than the baselines in a statistically significant way.

-- it would have been nice to get some insight for why DRL-abs does not always perform better than DRL, as well as how these models were trained/how their parameters were validated. Could the authors comment on this?

-- why can VIPER not solve the generalization task? If it’s also operating on the abstract space, I would have thought it should be invariant to the grid size since it will only have information about objects in the environment? Similarly why did the authors only evaluate generalization on StairClimber and Maze? It should be trivial to expand all of the problems to larger grids and it seems a bit weird to pick only 2.

-- a simple way to keep the observation size the same for DRL between different problem sizes, which would also be a more fair comparison to the LEAPS model, would be to use a fully connected network (Long*, Shelhamer*, Darrell, 2015) as the policy, as that will look at grid locations in a more local way and be invariant to problem size (see Silver, 2020 who similarly found that this type of network significantly out-performed a fully connected model for changing grid sizes). This might perform much better than the current method while also generalizing better, and be a good test of whether it’s the programmatic structure of the policy that is required, or if it’s more of a “locality” or “relational” requirement.

-- I think details about the RL aspect of the test tasks needs to be made very clear. E.g. how many episodes does an agent get within each task? How many training tasks do the DRL agents receive before they are evaluated? How are the training tasks for these distinguished from the test tasks for generating the numbers in Table 3?

Discussion
++ excellent pointers to limitations of the work - a crucial component of ML papers.
-- I would have liked to see a little more discussion on why programs specifically vs. hierarchical policies vs. abstract action spaces, and how this kind of approach would be expected to work for problems that are not specifically designed for program synthesis techniques. However I recognize there isn’t too much space and the authors alluded to this in the results sections.
 Yes they have addressed these adequately.

 Thank you to the reviewers for the detailed response addressing all of my main concerns. I have updated my score to 7 to reflect this, and thank the reviewers for taking the time to incorporate improvements based on my comments, and for the thoroughness and scientific rigor that they have demonstrated in the response.


 We thank the reviewer for the thorough and constructive comments. Please find the response to your questions below:

Q: Figure 2 caption

A: We thank the reviewer for pointing this out. We have revised the caption of Figure 2 to make it more self-contained by explaining all the notations in detail.

&nbsp;

Q: Other domains

A: The reviewer is absolutely correct. Verifying if LEAPS can produce longer and more complex programs in domains that can induce more complex and compositional behaviors, such as Minecraft and CARLA, is our immediate future work.

&nbsp;

Q: Other program synthesis baselines

A: Many program synthesis works study synthesizing Karel programs. While [1-3] learn to synthesize Karel programs from input/output pairs (i.e. input/output Karel states), [4-6] aim to infer Karel programs from videos/execution traces/Karel trajectories. However, since these works do not synthesize programs purely from reward signals to solve tasks described by MDPs, they are not directly comparable to our work.

[1] “Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis” in ICLR 2018

[2] "Improving Neural Program Synthesis with Inferred Execution Traces" in NeurIPS 2018

[3] "Execution-Guided Neural Program Synthesis" in ICLR 2019

[4] "Neural Program Synthesis from Diverse Demonstration Videos" in ICML 2018

[5] "Watch, Reason and Code- Learning to Represent Videos Using Program" in ACM Multimedia 2019

[6] "PLANS: Neuro-Symbolic Program Learning from Videos" in NeurIPS 2020

&nbsp;

Q: Sampling dataset programs

A: As noted by the reviewer, generating programs by uniformly sampling all the tokens from the DSL would yield programs that mainly only contain action tokens that are “not interesting” since the chance to synthesize conditional statements with correct grammar is lower. Therefore, to produce programs that are longer and deeply nested with conditional statements to induce more complex behaviors, we propose to sample programs using a probabilistic sampler. Specifically, we design an STMT_STMT (s := s1; s2 in Figure 1) token that breaks up the current token into two, each of which is resampled according to the same token distribution. By sampling the STMT_STMT and conditional tokens (WHILE, IF, IFELSE) with higher probabilities as mentioned in Table 5, we are able to obtain programs that induce more complex and longer behaviors. More details on program dataset generation with a probabilistic sampler are described in Appendix Section A.

&nbsp;

Q: Program embedding space smoothness on the test set

A: We thank the reviewer for the suggestion. We have performed the program embedding space smoothness evaluation on LEAPS and its ablations with the test dataset programs and the result is as follows. The same trend from the train/val sets applies to the test set. We have revised the paper to incorporate this result.

|            | LEAPS-P | LEAPS-P+R | LEAPS-P+L | LEAPS |
| ---------- | ------- | --------- | --------- | ----- |
| Training   | 0.22    | 0.22      | 0.31      | 0.31  |
| Validation | 0.22    | 0.21      | 0.27      | 0.27  |
| Testing    | 0.22    | 0.22      | 0.28      | 0.27  |

&nbsp;

Q: $R_{mat}$ reward and Table 2

A: We apologize for the confusion. For one program pair, $R_{mat}$ should actually be defined as the percent of times traces match (from the beginning of the trace), not the number of times they match---we thank the reviewer for pointing this out. We have modified this by adding an $\frac{1}{N}$ to $R_{mat}$’s definition.

In Table 2, we calculate $R_{mat}$ between one program from the dataset and its k=10 nearest neighbor programs and take the average. Table 2 represents the average $R_{mat}$ averaged over the entire dataset. We have revised the paper to include these details in the caption of Table 2.

&nbsp;

Q: Equation 4 - latent behavior reconstruction loss definition

A: We thank the reviewer for pointing this out and we have revised Eq. 4 accordingly. Specifically, we have changed $EXEC_t (\rho)$ to $I\\{EXEC_i (\hat{\rho}) == EXEC_i (\rho) \\}$, where $t \rightarrow i$ so that it changes with the inner sum index, and $I\{\}$ represents the identity function so that it becomes a 1 in the case of the executed action and 0 otherwise.

&nbsp;

Q: Figures in the Appendix

A: We thank the reviewer for pointing this out. We have revised the paper so that all the Figures and Tables that are not presented in the main paper but the Appendix are mentioned as “Figure/Table X (in Appendix)“.


 We thank the reviewer for the thorough and constructive comments. Please find the response to your questions below:

Q: Decision trees handling loops

A: We thank the reviewer for pointing this out with a detailed description and a reference. As noted by the reviewer, the scanning function in Silver et al, AAAI 2020 certainly can produce looping behavior (with less expressiveness, as the reviewer also points out). We have made the following changes in the revised paper to account for  this:
- The statements in the introduction and the related work are toned down.
- A detailed explanation of how decision trees can handle loops is added to the related work section with a reference to “Few-Shot Bayesian Imitation Learning with Logical Program Policies” in AAAI 2020.

&nbsp;

Q: Missing related work

A: We thank the reviewer for suggesting the missing related works and have revised the paper to properly discuss them, including:
- “LoopDISTILL: Learning Looping Domain-Specific Planners from Example Plans” in ICML 2003
- “Learning Teleoreactive Logic Programs from Problem Solving” in ILP 2005

&nbsp;

Q: Program synthesis baselines other than VIPER

A: We selected VIPER as it is the most directly comparable baseline because it also does not need additional supervision and can be readily applied to all of our tasks with no extra assumptions over our method. In contrast, REPL and [1-3] rely on program input/output pairs to synthesize a program, and [4-6] require entire program execution traces. Also, [7] requires program templates and [8] lacks starter code.

[1] “Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis” in ICLR 2018

[2] "Improving Neural Program Synthesis with Inferred Execution Traces" in NeurIPS 2018

[3] "Execution-Guided Neural Program Synthesis" in ICLR 2019

[4] "Neural Program Synthesis from Diverse Demonstration Videos" in ICML 2018

[5] "Watch, Reason and Code- Learning to Represent Videos Using Program" in ACM Multimedia 2019

[6] "PLANS: Neuro-Symbolic Program Learning from Videos" in NeurIPS 2020

[7] “Programmatically interpretable reinforcement learning” in ICML 2018

[8] “Synthesizing programmatic policies that inductively generalize” in ICLR 2020


&nbsp;

Q: Insignificant results of program reconstruction

A: We agree with the reviewer that standard deviations are high in Table 1 where we compare LEAPS to its ablations. Therefore, to justify the proposed method, we also provide qualitative results showing the programs produced by LEAPS to its ablations in Appendix Figure 7, which shows that LEAPS is able to synthesize programs that are more structured and concise.

&nbsp;

Q: DRL vs. DRL-abs

A: We hypothesize that DRL-abs does not always outperform DRL due to imperfect perception (i.e. state abstraction) design. DRL-abs takes abstract states as input (i.e. frontIsClear, leftIsClear, rightIsClear, markerPresent in our design), which only describe local perception while omitting the information of the entire map. Therefore, for tasks such as StairClimber, Harvester, and CleanHouse, which would be easier to solve with access to the entire Karel grid, DRL might outperform DRL-abs. In this work, DRL-abs’ abstract states are the perceptions from the DSL we synthesize programs with to make the comparisons fair against our method as well as analyzing the effects of abstract states in the DRL domain. However, a more sophisticated design for perception/state abstraction could potentially improve the performance of DRL-abs. 

To validate the performance of DRL and DRL-abs baselines, we perform a grid search over 72 hyperparameter configurations, with 5 seeds each, and report the results of the configuration with the highest final reward after training when averaged over all seeds (see Appendix Section F.1).

&nbsp;

Q: VIPER generalization

A: As the reviewer mentions, VIPER is operating on the abstract state space which is invariant to grid size. However, for the reasons below, it is still unable to transfer the behavior to the larger grid despite its abstract state representation. We hypothesize that VIPER’s performance suffers on zero-shot generalization for two main reasons.
- It is constrained to imitate the DRL teacher policy during training, which is trained on the smaller grid sizes. Thus its learned policy also experiences difficulty in zero-shot generalization to larger grid sizes.
- Its decision tree policies cannot represent certain looping behaviors as they simply perform a one-to-one mapping from abstract state to action, thus making it difficult to learn optimal behaviors that require a one-to-many mapping between an abstract state and a set of desired actions. Empirically, we observed that training losses for VIPER decision trees were much higher for tasks such as StairClimber which require such behaviors. 

&nbsp;

Q: Fully Convolutional Networks (FCN) 

A: As the reviewer points out, it is possible that a DRL policy employing a fully convolutional network (FCN) can handle varying observation sizes. However, we hypothesize that the generalization performance will still be poor as there is a large increase in the number of features that the FCN architecture needs to aggregate when transferring from 8x8/12x12 state inputs to 100x100 inputs---a \~10x input size increase that FCN is not specifically designed to deal with. 

Upon the request of the reviewer, we have included its results in the generalization table listed in the response to the next question. DRL-FCN, where we have replaced the policy and value function networks of PPO with an FCN, does manage to perform zero-shot transfer marginally better than DRL on Maze and Harvester. However, it obtains a negative reward on StairClimber as it attempts to navigate away from the stairs when transferring to the 100x100 grid size. Its performance is still far worse than LEAPS and VIPER on most tasks, demonstrating that the programmatic structure of the policy is important for these tasks.

We note that we performed a hyperparameter search for DRL-FCN to ensure the performances on the original grid size were sufficient before transferring. The table below compares the performance of DRL-FCN to DRL on tasks at their standard grid sizes:

|           | StairClimber | Maze        | FourCorner  | TopOff      | Harvester   |
|-----------|--------------|-------------|-------------|-------------|-------------|
| DRL       | 1.00 (0.00)  | 1.00 (0.00) | 0.29 (0.05) | 0.32 (0.07) | 0.49 (0.07) |
| DRL-FCN | 1.00 (0.00) | 0.97 (0.03) | 0.20 (0.34) | 0.28 (0.12) | 0.46 (0.16) |

&nbsp;

Q: Additional generalization experiments (FourCorner, TopOff, Harvester)

A: To evaluate zero-shot generalization performance, it would require methods to work reasonably well on the original tasks. Therefore, we only considered StairClimber and Maze for generalization experiments because most methods achieve reasonable performance on these two tasks, with DRL and LEAPS both solving these tasks fully and DRL-abs solving Maze fully.

Upon the request of the reviewer, we have performed additional comparisons on FourCorner, TopOff, and Harvester (excluding CleanHouse because only LEAPS has a meaningful level of performance on it). The result is summarized in the table below.

|           | StairClimber | Maze        | FourCorner  | TopOff      | Harvester   |
|-----------|--------------|-------------|-------------|-------------|-------------|
| DRL       | 0.00 (0.00)  | 0.00 (0.00) | 0.00 (0.00) | 0.01 (0.01) | 0.00 (0.00) |
| DRL-FCN | -0.20 (0.10) | 0.01 (0.01) | 0.00 (0.00) | 0.01 (0.01) | 0.02 (0.00) |
| DRL-abs   | 0.00 (0.00)  | 0.04 (0.05) | 0.37 (0.44) | 0.15 (0.12) | 0.02 (0.01) |
| VIPER     | 0.00 (0.00)  | 0.10 (0.12) | 0.40 (0.42) | 0.03 (0.00) | **0.04 (0.00)** |
| LEAPS     | **1.00 (0.00)** | **1.00 (0.00)** | **0.45 (0.37)** | **0.21 (0.03)** | 0.00 (0.00) |

This result suggests that LEAPS generalizes well on FourCorner and maintains the best performance on TopOff. All the methods perform poorly on Harvester. In summary, LEAPS performs the best on 4 out of these 5 tasks, further demonstrating its superior zero-shot generalization performance, which aligns with the finding in the original paper. We have revised the paper to include this result and the discussion.


 We thank the reviewer for the thorough and constructive comments. Please find the response to your questions below:

Q: Neural program executor - architecture and evaluation

A: The neural program executor takes a latent program vector, current state, previous action as input and predicts a sequence of actions that execute the program. Since it does not deal with explicitly written programs structured in a DSL, we simply employ a GRU-based recurrent neural network to approximate the neural execution.

To evaluate how close the predicted neural execution traces are to the execution traces of the ground-truth programs, we consider the following metrics: 
Action token accuracy: the percentage of matching actions in the predicted execution traces and the ground-truth execution traces.
Action sequence accuracy: the percentage of matching action sequences in the predicted execution traces and the ground-truth execution traces. It requires that a predicted execution trace entirely matches the ground-truth execution trace.
LEAPS’ neural program executor achieves an action token accuracy of 96.5% and an action sequence accuracy of 91.3% when the learning converges. We have revised the paper to incorporate this information in the Appendix.

&nbsp;

Q: Behavioural losses - predict action-state sequences

A: As noted by the reviewer, in general, learning to predict action-state sequences can potentially provide a stronger supervision signal when optimizing the behavioral losses. Yet, in this work, we consider an environment with a deterministic transition function. Therefore, for calculating the program reconstruction loss (Eq. 2), matching program action trajectories is equivalent to matching state-action trajectories.

To calculate the latent reconstruction loss (Eq. 4), we leverage a set of different trajectories with randomly sampled starting states for each program. Thus we need a policy conditioned on at least the first state. We found this prediction task to be difficult as is, requiring teacher forcing during training and a recurrent policy, so additionally predicting the subsequent Karel environment grid states seemed to yield limited improvement in this case.

We thank the reviewer for the suggestion. We have revised the paper to incorporate this discussion into Section F in the Appendix.

&nbsp;

Q: Latent space interpolations

A: We thank the reviewer for this suggestion. It would definitely be informative to traverse the learned program embedding space and visualize interpolations between different programs. 
To learn a program embedding space that allows for smooth interpolation, we propose three sources of supervision. We aim to verify the effectiveness of it, as suggested by the reviewer, by 
1) Sampling a pair of programs from the dataset (START program  and END program below)
2) Encoding the two programs into the learned program embedding space
3) Linearly interpolating between the two latent programs to obtain a number of (eight) interpolated latent programs
4) Decoding the latent programs to obtain interpolated programs (program 1 to program 8 below)

The results of several pairs of programs have been added to the revised Appendix. Here, we show two pairs of programs and their interpolations in between below as examples. Specifically, the first pair of programs are closer to each other in the latent space and the second pair of programs are further from each other. We observe that the interpolations between the closer program pair exhibit smoother transitions and the interpolations between the further program pair display more dramatic change. 
 
Programs that are close to each other in the learned program embedding space and the interpolations in between:
 
START program: `DEF run m( turnRight move WHILE c( frontIsClear c) w( move w) WHILE c( not c( frontIsClear c) c) w( move w) IF c( frontIsClear c) i( move i) m)`
 
program 1: `DEF run m( turnRight move WHILE c( frontIsClear c) w( move w) WHILE c( not c( frontIsClear c) c) w( move w) IF c( frontIsClear c) i( move i) m)`
 
program 2: `DEF run m( turnRight move WHILE c( frontIsClear c) w( move w) IF c( not c( frontIsClear c) c) i( move i) m)`
 
program 3: `DEF run m( turnRight move WHILE c( frontIsClear c) w( move w) IF c( not c( frontIsClear c) c) i( move i) m)`
 
program 4: `DEF run m( turnRight move WHILE c( frontIsClear c) w( move w) IF c( not c( frontIsClear c) c) i( move i) m)`
 
program 5: `DEF run m( turnRight move WHILE c( frontIsClear c) w( move w) IF c( not c( frontIsClear c) c) i( move i) m)`
 
program 6: `DEF run m( turnRight move WHILE c( frontIsClear c) w( move w) IF c( not c( frontIsClear c) c) i( move i) m)`
 
program 7: `DEF run m( turnRight move turnLeft WHILE c( frontIsClear c) w( move w) IF c( not c( frontIsClear c) c) i( putMarker i) m)`
 
program 8: `DEF run m( turnRight move turnLeft WHILE c( frontIsClear c) w( move w) IF c( not c( frontIsClear c) c) i( putMarker i) m)`
 
END program: `DEF run m( turnRight move turnLeft WHILE c( frontIsClear c) w( move w) IF c( not c( frontIsClear c) c) i( putMarker i) m)`
 
Programs that are far from each other in the learned program embedding space and the interpolations in between:

START program: `DEF run m( turnRight turnLeft turnLeft move turnRight putMarker move m)`

program 1: `DEF run m( turnRight turnLeft turnLeft move turnRight putMarker move m)`

program 2: `DEF run m( turnRight turnLeft turnLeft move WHILE c( frontIsClear c) w( putMarker w) turnRight move m)`

program 3: `DEF run m( turnRight turnLeft move turnLeft WHILE c( frontIsClear c) w( putMarker w) move m)`

program 4: `DEF run m( turnRight turnLeft move WHILE c( frontIsClear c) w( turnLeft w) IF c( not c( frontIsClear c) c) i( move i) m)`

program 5: `DEF run m( turnRight move turnLeft WHILE c( frontIsClear c) w( move w) IF c( not c( frontIsClear c) c) i( putMarker i) m)`

program 6: `DEF run m( move turnRight turnLeft move WHILE c( frontIsClear c) w( IF c( not c( rightIsClear c) c) i( putMarker i) w) m)`

program 7: `DEF run m( move turnRight turnLeft move WHILE c( frontIsClear c) w( IF c( not c( rightIsClear c) c) i( turnLeft i) w) m)`

program 8: `DEF run m( move turnRight move WHILE c( frontIsClear c) w( IF c( not c( rightIsClear c) c) i( turnLeft i) w) m)`

END program: `DEF run m( move turnRight move WHILE c( frontIsClear c) w( IF c( not c( rightIsClear c) c) i( turnLeft i) w) m)`

&nbsp;

Q: Computational resources and time

A: The following two machines were used to run the experiments (as reported in Section G):
- M1: NVIDIA Titan Xp with 40 virtual CPU cores
- M2: NVIDIA GeForce RTX 2080 Ti with 72 virtual CPU cores

The experiment cost (GPU memory/time) is as follows:
- Learning Program Embedding Stage
    - LEAPS-P:  4.2GB/13hrs on either M1 or M2
    - LEAPS-P+R: 4.2GB/\~44-54hrs on M2
    - LEAPS-P+L: 8.7GB/26hrs on either M1 or M2
    - LEAPS: 8.8GB/104hrs on M1, 8.8GB/58hrs on M2
- Latent Program Search Stage
    - CEM search: 0.8GB/4\~10min (depends on the CEM population size and the number of iterations until convergence) 
- DRL/DRL-abs/DRL-abs-t
    - 1-2GB/\~1hr per run with parallelization across 10 processes
- HRL/HRL-abs
    - 1-2.5GB/\~2.5hrs per run
- VIPER
    - 1GB/20\~30 minutes (excluding the time for learning its teacher policy)

We have added the experiment cost details to Section G in the Appendix.


 Q: RL details

A: We thank the reviewer for pointing out that some of the RL training details were not that clearly organized. The DRL agents receive 1M timesteps (\~120k episodes) of training before evaluation as their performance converges within 1M steps for all tasks and methods (see Appendix Section F.1). 

Regarding the reviewer’s question about differences between the training/test tasks, please note that we are not assuming a multi-task or train-then-transfer setup for the main Karel tasks. For all the six task MDPs (i.e. StairClimber, FourCorner, TopOff, Maze, CleanHouse, and Harvester) considered in Table 3, all the methods are trained and evaluated on the same MDP (task), except for DRL-abs-t; it is specifically designed to be the train-then-transfer baseline method (i.e. trained on other tasks, and then fine-tuned and evaluated on the target task).

For example, a DRL StairClimber policy is trained with different agent/goal locations and tested on the same distribution of agent/goal locations where no generalization is required (exact details for all environments are listed in Appendix Section E). Thus, the training and the testing task is identical.

When either training or evaluating, we randomly sample initial configurations for the DRL agents upon every episode reset. We have updated the paper to make these environment details more clearly integrated by explicitly stating them in the main text in Section 5.1.

&nbsp;

Q: Programs specifically vs. hierarchical policies vs. abstract action spaces

A: As pointed out by the reviewer, programmatic inductive bias, hierarchical structure of policies, and abstract action and state spaces play different roles in boosting the performance. In this work, we mainly study how structuring a policy in a formal language could lead to better performance and could yield interpretability and generalization ability. To justify the proposed method with fair comparisons, we try to differentiate the benefit coming from programmatic inductive bias (LEAPS’ ablations and VIPER), hierarchical structure of policies (HRL, HRL-abs), and abstract state space (DRL-abs, DRL-abs-t, HRL-abs) by considering a variety of baselines.


 We thank the reviewer for the thorough and constructive comments. Please find the response to your questions below:

Q: The errors in the DSL definition

A: We thank the reviewer for pointing out these errors. We have revised the paper to incorporate the suggestions regarding adding the correct bracketing function, changing “percept” to “percept h”, condition’s definition to “percept h | not percept h”, and the “while” definition into the DSL figure. We appreciate the detailed and well-thought-out suggestions from the reviewer.

&nbsp;

Q: Pointer to the appendix - the architecture of the program encoder and decoder

A: We thank the reviewer for this suggestion. We have revised the paper to add this pointer to the end of the first paragraph of Section 4.1.

&nbsp;

Q: Syntax checker

A: We restrict our decoder from predicting syntactically invalid programs by masking out tokens that make a program syntactically invalid at each timestep. Specifically, we improve upon the syntax checker proposed in [1]. This syntax checker is designed as a state machine that keeps track of a set of valid next tokens based on the current token, open code blocks ((\texttt{while, if, ifelse})) in the given partial program and grammar rules of our DSL. Since we generate a program as a sequence of tokens, the syntax checker outputs at each timestep a mask $M = \\{ \-\infty, 0 \\}^{|\ number\ \ of\ \ tokens\ \ in\ \ DSL\ |} \   where\ M_{j} = \-\infty$ if the j-th token is not a valid token in the current context, and 0 otherwise. This mask is added to the output of the last layer of the decoder, just before the Softmax operation that normalizes the output to a probability over the tokens. We have updated Appendix Section F6 to incorporate these details.

[1] “Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis” in ICLR 2018

&nbsp;

Q: Sampling dataset programs - procedure, meaningful behaviors, and termination

A: Indeed, simply generating programs by uniformly sampling all the tokens from the DSL would yield programs that mainly only contain action tokens that do not carry out meaningful behaviors since the chance to synthesize conditional statements with correct grammar is lower. Therefore, to produce programs that are longer and deeply nested with conditional statements to induce more complex behaviors, we propose to sample programs using a probabilistic sampler. Specifically, we utilize an STMT_STMT token (s := s1; s2 in Figure 1) that breaks up the current token into two, each of which is resampled according to the same token distribution.  By sampling the STMT_STMT and conditional tokens (\texttt{while, if, ifelse}) with higher probabilities as mentioned in Appendix Table 5, we are able to obtain programs that induce more complex and longer behaviors. More details on program dataset generation with a probabilistic sampler can be found in Appendix Section A. We have also revised the section to include the above intuition.

As noted by the reviewer, these programs are not guaranteed to terminate. Therefore, we allow a maximum execution length of 100 timesteps in the environment for computing the Program Behavior and Latent Reconstruction losses. We have added this detail to Appendix Section A.

&nbsp;

Q: Typos

A: We thank the reviewer for pointing out the errors. We have fixed them in the revised paper.

 Q: Program evolution - how the best found program evolves over the course of searching

A: We thank the reviewer for this suggestion. It definitely would be informative to see how predicted programs evolve over the course of searching. For the requested loss/reward evolution over the search, we have visualized converged CEM search trajectories and the reward each program gets on the StairClimber task in Appendix Figure 15. Upon the request of the reviewer, we also present the corresponding predicted programs below. We have revised the paper to include this in Section J.

Predicted programs evolving during the latent program search stage on the StairClimber task:

Iteration: 1 - `DEF run m( IF c( frontIsClear c) i( pickMarker i) WHILE c( leftIsClear c) w( move w) IFELSE c( frontIsClear c) i( turnRight move i) ELSE e( move e) m)`

Iteration: 2 - `DEF run m( WHILE c( markersPresent c) w( move w) IFELSE c( frontIsClear c) i( turnLeft i) ELSE e( move e) WHILE c( leftIsClear c) w( move w) m)`

Iteration: 3 - `DEF run m( WHILE c( not c( frontIsClear c) c) w( move turnRight w) WHILE c( leftIsClear c) w( turnLeft move w) m)`

Iteration: 4 - `DEF run m( WHILE c( not c( frontIsClear c) c) w( pickMarker move w) WHILE c( leftIsClear c) w( turnLeft move w) m)`

Iteration: 5 - `DEF run m( WHILE c( not c( frontIsClear c) c) w( pickMarker turnRight w) WHILE c( leftIsClear c) w( move turnLeft w) m)`

Iteration: 6 - `DEF run m( WHILE c( not c( frontIsClear c) c) w( pickMarker turnRight w) WHILE c( leftIsClear c) w( move turnLeft w) m)`

Iteration: 7 - `DEF run m( WHILE c( not c( leftIsClear c) c) w( turnRight w) IFELSE c( frontIsClear c) i( move i) ELSE e( turnLeft e) WHILE c( rightIsClear c) w( move w) m)`

Iteration: 8 - `DEF run m( WHILE c( not c( leftIsClear c) c) w( turnRight move w) WHILE c( markersPresent c) w( turnLeft move w) m)`

Iteration: 9 - `DEF run m( WHILE c( not c( noMarkersPresent c) c) w( turnRight move w) WHILE c( not c( frontIsClear c) c) w( turnLeft move w) m)`

Iteration: 10 - `DEF run m( WHILE c( not c( noMarkersPresent c) c) w( turnRight move w) WHILE c( leftIsClear c) w( turnLeft move w) m)`

Iteration: 11 - `DEF run m( WHILE c( not c( leftIsClear c) c) w( turnRight move w) WHILE c( noMarkersPresent c) w( turnLeft move w) m)`

Iteration: 12 - `DEF run m( WHILE c( not c( leftIsClear c) c) w( turnRight move w) WHILE c( noMarkersPresent c) w( turnLeft move w) m)`

Iteration: 13 - `DEF run m( WHILE c( not c( leftIsClear c) c) w( turnRight move w) WHILE c( noMarkersPresent c) w( turnLeft move w) m)`

Iteration: 14 - `DEF run m( WHILE c( not c( markersPresent c) c) w( turnRight move w) WHILE c( rightIsClear c) w( move turnLeft w) m)`

Iteration: 15 - `DEF run m( WHILE c( not c( markersPresent c) c) w( turnRight move w) WHILE c( rightIsClear c) w( move turnLeft w) m)`

Iteration: 16 - `DEF run m( WHILE c( not c( markersPresent c) c) w( turnRight move w) WHILE c( rightIsClear c) w( move turnLeft w) m)`

Iteration: 17 - `DEF run m( WHILE c( not c( markersPresent c) c) w( turnRight move w) WHILE c( rightIsClear c) w( move turnLeft w) m)`

Iteration: 18 - `DEF run m( WHILE c( not c( markersPresent c) c) w( turnRight move w) WHILE c( rightIsClear c) w( move turnLeft w) m)`

Iteration: 19 - `DEF run m( WHILE c( not c( markersPresent c) c) w( turnRight move w) WHILE c( rightIsClear c) w( move turnLeft w) m)`

Iteration: 20 - `DEF run m( WHILE c( not c( markersPresent c) c) w( turnRight move w) WHILE c( rightIsClear c) w( move turnLeft w) m)`

Iteration: 21 - `DEF run m( WHILE c( not c( markersPresent c) c) w( turnRight move w) WHILE c( rightIsClear c) w( move turnLeft w) m)`

Iteration: 22 - `DEF run m( WHILE c( not c( markersPresent c) c) w( turnRight move w) WHILE c( rightIsClear c) w( move turnLeft w) m)`

Converged - `DEF run m( WHILE c( not c( markersPresent c) c) w( turnRight move w) WHILE c( rightIsClear c) w( turnLeft move w) m)`

We observe that the sampled programs above consistently improve as the number of iterations increases, justifying the effectiveness of the learned program embedding and the CEM search.

&nbsp;

Q: CEM phenomenon in Figure 15

A: We apologize for the possible confusion. Figure 15 visualizes the trajectory of a converged CEM search, showing that the rewards consistently increase and the CEM population gradually moves toward where the ground-truth program is located. Yet, the populations might not always exactly converge to where the ground-truth latent program is. We hypothesize this could be attributed to the following reasons:
- CEM convergence: while the CEM search converges, it can still be suboptimal. Since the search terminates when the next candidate latent program obtains the maximum reward (1.1 as shown in the figure) for 10 iterations, it might not exactly converge to where a ground-truth program is. 
- Dimensionality reduction: we visualized the trajectories and programs by performing dimensionality reduction from 256 to 2 dimensions with PCA, which could cause visual distortions.
- Suboptimal learned program embedding space: while we aim to learn a program embedding space where all the programs inducing the same behaviors are mapped to the same spot in the embedding space, it is still possible that programs that induce the desired behavior can distribute to more than one location in a learned program embedding space. Therefore, CEM search can converge to somewhere that is different from the ground-truth latent program.
We have revised Appendix Section J accordingly to include this discussion.

&nbsp;

Q: Program complexity

A: The reviewer is absolutely correct. Verifying if LEAPS can apply to produce longer and more complex programs in domains that can induce more complex and compositional behaviors, such as Minecraft and CARLA, is our immediate future work.

&nbsp;

Q: Robotics applications

A: As pointed out by the reviewer, one way to make the LEAPS framework applicable to robotics domains would be simultaneously learning perception modules and action controllers. Other possible solutions include incorporating program execution methods [1-3] that are designed to allow program execution or designing DSLs that allow pre-training of perception modules and action controllers. We have revised the paper to incorporate this discussion in the Appendix.

[1] “Program Guided Agent” in ICLR 2020

[2] “Reinforcement Learning of Implicit and Explicit Control Flow in Instructions” in ICML 2021 

[3] “Program Synthesis Guided Reinforcement Learning” in arXiv 2021 

 We would like to sincerely thank all the reviewers for their thorough and constructive comments. We have revised our paper to incorporate them. The major changes are summarized as follows.

[Additional experiments] We have conducted the following experiments to better answer the questions asked by the reviewers. The results and the corresponding discussions have been included in the revised paper.
- Interpolations in the learned program embedding space (Reviewer dyBV)
- Program evolution - how the best found program evolves over the course of searching (Reviewer dyBV)
- DRL policy employing a fully convolutional network (Reviewer gcpQ)
- Additional generalization experiments (FourCorner, TopOff, Harvester) (Reviewer gcpQ)
- Program embedding space smoothness on the test set (Reviewer etfX)

[Details] We revised the main paper and the Appendix to clearly organize the information related to the topics listed below.
- Neural program executor (Reviewer dyBV)
- Computational resources and time (Reviewer dyBV)
- DRL training and evaluation (Reviewer gcpQ)
- Sampling dataset programs (Reviewer FK1E and Reviewer etfX)
- Syntax checker (Reviewer FK1E)

[Related work] We thank the reviewers for pointing out relevant works that are missing and we have revised the related work section to include the following references and properly discuss them:
- “Few-Shot Bayesian Imitation Learning with Logical Program Policies” in AAAI 2020 (Reviewer gcpQ)
- “LoopDISTILL: Learning Looping Domain-Specific Planners from Example Plans” in ICML 2003 (Reviewer gcpQ)
- “Learning Teleoreactive Logic Programs from Problem Solving” in ILP 2005 (Reviewer gcpQ)

[Intuitions] We have revised the paper to include additional discussions on the following topics.
- CEM phenomenon in Figure 15 (Reviewer dyBV)
- DRL vs. DRL-abs (Reviewer gcpQ)
- VIPER generalization (Reviewer gcpQ)

[Discussions] We have addressed the suggestions and questions from the reviewers
- Behavioural losses - predict action-state sequences (Reviewer dyBV)
- Results of program reconstruction (Reviewer gcpQ)
- Programs specifically vs. hierarchical policies vs. abstract action spaces (Reviewer gcpQ)

[Writing] As suggested by the reviewers, we have improved the writing by:
- Add a pointer to the appendix for the architecture of the program encoder and the decoder. (Reviewer FK1E)
- Figure 2 caption: make it more self-contained by explaining all the notations in detail. (Reviewer etfX)
- Figures and Tables in the Appendix are mentioned as “Figure/Table X (in Appendix)“. (Reviewer etfX)

[Errors and unclear parts] We have revised the paper to fix the errors/unclear parts pointed out by the reviewers.
- The errors in the DSL definition (Reviewer FK1E)
- Typos (Reviewer FK1E)
- $R_{mat}$ reward and Table 2 (Reviewer etfX)
- Equation 4 - latent behavior reconstruction loss definition (Reviewer etfX)


This paper presents a two-stage method for reinforcement learning via program synthesis, and shows that is is effective compared to baselines for a number of simple gridworld tasks, deminstrating a degree of out-of-distribution generalisation. The first stage of the process learns a latent space for encoding programs, and the second stage searches that latent space for programs that maximise reward in  the task. Overall a very nice paper that deserves publication.  This is a very substantial and impressive piece of work. Program synthesis is a very promising avenue for building RL agents that achieve better generalisation, and whose representations are humanly interpretable. A major accomplishment of this paper is to learn embeddings of programs (in a variant of the Karel language) using a VAE with a recurrent encoder and decoder and three different losses. (I’m not sufficiently famiiar with recent ML literature on program synthesis to know whether this is wholly new - so I rely on other reviewers to confirm this - but it seems like a significant step to me.) The authors carry out an extensive evaluation, with careful ablations, and show the effectiveness of the approach on a number of (admittedly quite simple) RL tasks. The most important result, to my mind, comes towards the end of the main text, which is to show that learned programs generalise to out-of-distribution grid sizes, where other approaches fail. The paper is clearly written and logically organised, although the method is complex, which means that a lot of referring to the Appendix is needed to follow it. (This is not a criticism, of course; it’s just a corrolary of the page limit.) Extensive material is supplied in the Appendix to back up the main paper.

There are quite a few issues in the definition of the DSL in Figure 1.

1) “while(b)” should be “while(b): s”

2) the definition of “Perception” is, at present, unused. I suspect the authors meant to write “Percept h” rather than “Perception h”, and to write “percept h” in the definition of a “Condition”.

3) Since negation is the only logical operator, the definition of  “Condition” should perhaps be simply “percept | not percept”. (Or rather “percept h | not percept h”; see (2).) At present it allows for, say, “not not not not percept”, which probably unintended, and surely unnecessary.

4) Don’t you need some brackets in the grammar? Otherwise how do you distinguish “while(B): (S1 ; S2)” from “(while(B): S1) ; S2”? (Obviously this isn’t an issue in normal Karel, where white space is used to indicate structure.) Indeed, from the examples in the Appendix (Fig. 6), it’s clear there are more symbols used in the code than in the grammar of Fig. 1, such as “c(“ and “)c”, which fulfil this bracketing function, as described in the caption. It’s a minor thing, but I feel the DSL grammar in Fig. 1 should be made to match this; it’s really a syntactic variant of Karel.

Section 4.1: I was very curious, at this point, to know what the architecture of q_phi and p_theta was. It took me a while to hunt them down in the Appendix (Section F6). A pointer to this material in the main text (Section 4.1) would be helpful.

In the Appendix (Section F6), you state “we utilize a syntax checker … at the output of the decoder to limit predictions to syntactically valid tokens”. How do you do this? Does the training loop just skip to the next sample if a syntactically incorrect program is generated? This can’t be right? How do you use the syntax check?

How do you sample from the set of possible programs in order to learn the program embedding? Do they all carry out meaningful behaviours? Do they all terminate?


TYPOS

Lines 81-82: “This limit the set of synthesizable programs” -> “This limits the set of synthesizable programs”

Lines 108-109: “view this objective is a special case” -> “view this objective as a special case”

Figure 3 caption: “tasks that demands certain behaviors” -> “tasks that demand certain behaviors”

Line 166: “ through supervised instead of RL gradients” -> “ through supervised learning instead of RL gradients” (presumably)
 The authors include some material in the Discussion

- Authors use a two-stage method for program synthesis: (1) They
  create an embedding space capturing the semantics of the program
  space; (2) They search through the embedding space for a program
  that solves the task in hand.
- They use variational autoencoder to encode programs, where an
  encoder maps a sequence of program tokens into a latent space, while
  a decoder maps from the latent space into the original program. They
  use a combination of loss functions to ensure that not only
  syntactically similar programs are close to each other in the latent
  space but also semantically close programs. Loss function consists
  of a composition of 3 losses. (A) Classical beta VAE loss which
  helps that syntactically close programs are close to each other. (B)
  Program behavior reconstruction loss which helps that semantically
  similar programs are close to each other in the embedding space. For
  example, two syntactically different programs can make same actions
  under same situations. This loss is defined by difference between
  execution traces of original and reconstructed program. (C) Latent
  behavior reconstruction loss. This loss is constructed by learning a
  policy to predict correct action based on the latent program
  embedding. This allows for backpropagating gradient through the
  policy and the encoder to ensure better quality of the embeddings.
- After training the variational autoencoder, authors use a cross
  entropy method to find a program with maximal performance using the
  latent space. They sample the latent space distribution of programs,
  decode the latent program into an actual program, execute the
  program to obtain corresponding rewards, and repeat the process.
- Authors did a nice ablation study where they measured effectiveness
  of their individual design decisions: evaluating each of the loss
  function components, as well as the effectiveness of the latent
  space search method for finding the best performing program. They
  compared they method to other methods, deep reinforcement learning
  (DRL) based as well as Viper which is a decision tree policy
  obtained by mimicking a DRL agent.  - Originality: I find the approach to be novel, creatively designing
  loss functions ensuring quality latent space representation of
  programs.

- Significance: Program synthesis is an important and challenging
  research area, and thus I find the problem being solved
  important. Technique is illustrated on a simple (Karel) tasks which
  might seem insufficient, but the program synthesis field is still at
  the stage where small problems are hard to solve, thus I find the
  choice appropriate.
  
- Paper is well written and easy to follow. In addition, authors
  provide extensive appendix and accompanying web page with useful
  information.

- Figure 2 is very informative. I would suggest extending the caption
  of the figure so it is more self-contained. For example, explain
  what policy Pi is, as well as a1, a2, etc. I understood different
  parts of the figure only when I thoroughly read the paper.

- In the future, it would be good to evaluate the approach on tasks
  that are outside of the Karel domain.

- I do not have a good track of other approaches that can be used for
  synthesizing Karel programs, and thus am not certain if all relevant
  approaches are included in the comparison.

- Questions
    - Have you considered different ways of generating programs than
      pure random generation? Are there many programs that are 'not
      interesting' when performing random generation?
    - Program Embedding Space Smoothness. Is there a specific reason
      why embedding space search is performed on the train/val set, vs
      test set? Does the same trend apply to the test set?
    - In Table 2, do results represent percent of times that execution
      traces of two programs match (averaged across program pairs)? I
      understand that Rmat is defined as number of times traces match,
      not percent of times.
    - equation 4 (latent behavior reconstruction loss): Loss is
      referred to as a cross entropy loss. However, EXECt(ro) seems
      incorrect as it's constant in the inner sum. I assume it should
      be 1 in a case of the executed action, and 0 otherwise?

- Minor
    - Line 233: "shown in Figure 6". This figure is not in the main
      text. Authors did cover these areas. They could elaborate how does their approach scale to other domains.

Meta-review: This paper presents an interesting idea of learning policies for solving RL problems through program synthesis.  The key is to learn a good program embedding space, through 3 different losses that take care of 3 different aspects of a program representation.

The reviewers all appreciate the clarity of the paper and liked the idea.  The extensive ablations and supplementary material are also great for supporting the proposed approach.

In the future showing that this approach works on other domains beyond Karel, e.g. Minecraft and CARLA as suggested by the authors in the discussion, would be great.

Decision: accept