Title: Hierarchical Programmatic Option Framework
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning

Programmatic Reinforcement Learning

Hierarchical Reinforcement Learning

Program Synthesis
Abstract: Deep reinforcement learning aims to learn deep neural network policies to solve large-scale decision-making problems. However, approximating policies using deep neural networks makes it difficult to interpret the learned decision-making process. To address this issue, prior works (Trivedi et al., 2021; Liu et al., 2023; Carvalho et al., 2024) proposed to use human-readable programs as policies to increase the interpretability of the decision-making pipeline. Nevertheless, programmatic policies generated by these methods struggle to effectively solve long and repetitive RL tasks and cannot generalize to even longer horizons during testing. To solve these problems, we propose the Hierarchical Programmatic Option framework (HIPO), which aims to solve long and repetitive RL problems with human-readable programs as options (low-level policies). Specifically, we propose a method that retrieves a set of effective, diverse, and compatible programs as options. Then, we learn a high-level policy to effectively reuse these programmatic options to solve reoccurring subtasks. Our proposed framework outperforms programmatic RL and deep RL baselines on various tasks. Ablation studies justify the effectiveness of our proposed search algorithm for retrieving a set of programmatic options.

Hierarchical Programmatic Option Framework

Abstract

Deep reinforcement learning aims to learn deep neural network policies to solve large-scale decision-making problems. However, approximating policies using deep neural networks makes it difficult to interpret the learned decision-making process. To address this issue, prior works [10, 46, 74] proposed to use humanreadable programs as policies to increase the interpretability of the decision-making pipeline. Nevertheless, programmatic policies generated by these methods struggle to effectively solve long and repetitive RL tasks and cannot generalize to even longer horizons during testing. To solve these problems, we propose the Hierarchical Programmatic Option framework (HIPO), which aims to solve long and repetitive RL problems with human-readable programs as options (low-level policies). Specifically, we propose a method that retrieves a set of effective, diverse, and compatible programs as options. Then, we learn a high-level policy to effectively reuse these programmatic options to solve reoccurring subtasks. Our proposed framework outperforms programmatic RL and deep RL baselines on various tasks. Ablation studies justify the effectiveness of our proposed search algorithm for retrieving a set of programmatic options.

1 Introduction

Deep reinforcement learning (deep RL) has recently achieved tremendous success in various domains, such as controlling robots [26, 31], playing strategy board games [67, 68], and mastering video games [78, 82]. However, neural network policies learned by deep RL methods are not humaninterpretable, and their black-box nature poses challenges in scrutinizing model decisions and establishing user trust [45, 64]. Moreover, deep RL policies often suffer from overfitting and struggle to generalize to novel scenarios [17, 84], limiting their applicability in the context of most real-world applications.

To address these issues, programmatic RL frameworks [10, 46, 74] were proposed to represent policies as programs that detail task-solving procedures in a formal programming language. Particularly, Trivedi et al. [74] and Liu et al. [46] synthesize programs from continuous latent spaces, whereas Carvalho et al. [10] search programs directly from discrete programmatic spaces. Such program policies are human-readable and demonstrate significantly improved zero-shot generalizability from smaller state spaces to larger ones.

Despite encouraging results, prior programmatic RL frameworks are limited to generating concise programs that can only tackle short-horizon tasks. Particularly for long-horizon RL problems like robotic manipulation [2, 8] and autonomous driving [55, 58], these tasks consist of reoccurring subtasks and sparse rewards, necessitating a significant number of actions to be fully resolved. Therefore, the RL agent needs to learn a diverse and reusable set of skills to solve such tasks effectively.

Aiming to solve long and repetitive tasks with better policy interpretability, we borrow ideas from the option frameworks [4, 51, 66, 72] and latent-space-based programmatic RL frameworks [46, 74]. With that in mind, we propose the Hi erarchical P rogrammatic O ption framework (HIPO), which utilizes a set of programs with diverse skills as options (programmatic options) and learn a high-level policy to determine which programmatic option to be used based on the current state and the current option. By switching between these programmatic options, HIPO can reuse the skills encapsulated in these options to tackle long-horizon tasks with an arbitrary number of repeating subroutines.

Our framework contains three stages. (1) Constructing a program embedding space : To establish a program embedding space that smoothly and continuously parameterizes programs with diverse behaviors, we adopt the method proposed by Trivedi et al. [74]. (2) Retrieving a diverse set of effective and reusable programmatic options : We introduce a searching algorithm to retrieve a set of programmatic options from the learned program embedding space. Each programmatic option can be executed in the MDP and achieve satisfactory performance; more importantly, these programs are compatible and can be sequentially executed in any order. (3) Learning a high-level policy : To alter between a set of programmatic options, the high-level policy represented by neural networks takes the current environment state and the current programmatic option as input to predict the next programmatic option. This high-level policy can be learned using RL algorithms with the goal of maximizing the task return from the MDP.

To evaluate our proposed HIPO framework, we adopt the Karel domain [56], which characterizes an agent that navigates a grid world and interacts with objects. HIPO outperforms prior programmatic reinforcement learning and deep RL baselines on existing benchmarks [46, 74]. To further evaluate the performance and generalization ability to even longer horizons, we design a new set of tasks consisting of an arbitrary number of subtasks. Our framework shows better generalization in testing environments of different lengths. Ablation studies are also conducted to demonstrate the effectiveness of the proposed programmatic options retrieving process.

2 Related work

Program synthesis. Program synthesis techniques revolve around program generation to convert given inputs into desired outputs. These methods have demonstrated notable successes across diverse domains such as array and tensor manipulation [5, 22], string transformation [20, 29, 85], generating computer commands [44] and code [11, 42], graphics and 3D shape modeling [48, 73, 81], and describing agent behaviors [9, 13, 43, 69, 70]. Most program synthesis methods focus on task specifications such as input/output pairs, demonstrations, or language descriptions; in contrast, this work aims to synthesize human-readable programs as options to solve reinforcement learning tasks.

Programmatic reinforcement learning. Programmatic reinforcement learning methods [16, 80] explore structured representations for representing RL policies, including decision trees [7, 35], state machines [32], symbolic expressions [19, 33, 39, 49, 50, 83], and programs [1, 75, 76]. Liu et al. [46], Medeiros et al. [52], Moraes and Lelis [53], Trivedi et al. [74], and Carvalho et al. [10] attempted to produce policies described by domain-specific language programs to solve simple RL tasks. We aim to take a step toward addressing complex, long-horizon, repetitive tasks.

Hierarchical reinforcement learning. Hierarchical Reinforcement Learning (HRL) frameworks [4, 6, 40, 72, 77] focus on learning and operating across different levels of temporal abstraction, enhancing the efficiency of learning and exploration, particularly in sparse-reward environments. In this work, our proposed HIPO shares the same spirit with HRL frameworks [18, 21, 24, 25, 62, 63] that learn reusable skills as options. Instead of learning uninterpretable options as low-level policies, our framework aims to retrieve reusable and interpretable programs as options.

3 Problem formulation

Our goal is to devise a framework that generates a set of programs as options (low-level policies) and integrates them with high-level policies to tackle complex, long-term tasks defined by Markov Decision Processes (MDPs). To this end, we first synthesize a set of task-solving, diverse, compatible programs, then train a high-level policy to iteratively select and execute programs.

Domain specific language. This work adopts the domain-specific language (DSL) of the Karel domain [9, 13, 74], as illustrated in Figure 1. This DSL describes the control flows as well as the perception and actions of the Karel agent. Actions including move , turnRight , and putMarker define how the agent can interact with the environment. Perceptions, such as frontIsClear and markerPresent , formulate how the agent observes the environment. Control flows, e . g ., if , else , while , enable representing divergent and repetitive behaviors. Furthermore, Boolean and logical operators like and , or , and not allow for composing more intricate conditions. This work uses programs structured in this DSL to construct programmatic options.

Markov Decision Process (MDP). The tasks considered in this work can be formulated as finite-horizon discounted Markov Decision Processes (MDPs). The performance of HIPO is evaluated based on the execution traces of a series of programs (programmatic options) selected by its high-level policy. The rollout of a program ρ consists of a T -step sequence of state-action pairs { ( s t , a t ) } t =1 , ..., T obtained from a program executor EXEC ( · ) that executes program ρ to interact with an environment, resulting in the discounted return ∑ T t =0 γ t ( r t ) , where r t = R ( s t , a t ) denotes the reward function. We aim to maximize the total rewards by executing a series of programs following the high-level policy.

Program

ρ

:=

DEF run m

(

s

m

)

Repetition

n

:=

Number of repetitions

Perception

h

:=

frontIsClear

|

leftIsClear

|

rightIsClear

|

markerPresent

|

noMarkerPresent

Condition

b

:=

perception h

|

not perception h

Action

a

:=

move

|

turnLeft

|

turnRight

|

putMarker

|

pickMarker

Statement

s

:=

while c

(

b

c

)

w

(

s

w

)

|

s

1

;

s

2

|

a

|

repeat R=

n

r

(

s

r

)

|

if c

(

b

c

)

i

(

s

i

)

|

ifelse c

(

b

c

)

i

(

s

1

i

)

else e

(

s

2

e

)

Hierarchical Programmatic Option Framework. The proposed hierarchical framework consists of a set of programmatic options M = { m k } k =1 , ..., | M | as low-level policies and a high-level policy f ϕ that sequentially chooses one option at a time. Each option m i encapsulates a human-readable program ρ m i that will be executed if selected by the high-level policy f ϕ . On the other hand, the high-level policy f ϕ ( m i , s i T i ) outputs the probability distribution over all programmatic options M , given the last selected programmatic option m i at timestep T i and the current MDP state s i T i . If the next option m i +1 sampled from the distribution is the termination mode m term, the rollout will be terminated. Otherwise, the corresponding programmatic option ρ m i +1 will be executed and generates a sequence of state-action pairs { ( s i +1 t , a i +1 t ) } t =1 , ..., T i +1 before the high-level policy f ϕ selects the next programmatic option m i +2 .

4 Approach

We design a three-stage framework to search programmatic options and train a high-level policy represented by neural networks. The main goal is to maximize the return given a task described by an MDP. Firstly, as introduced in Section 4.1, we construct a program embedding space parameterizing programs smoothly and continuously with diverse behaviors. Then, Section 4.2 presents a method that retrieves a set of effective, diverse, and compatible programmatic options. Given retrieved programmatic options, Section 4.3 describes learning the high-level policy to determine probability distributions for options sampling. An overview of the proposed framework is illustrated in Figure 2.

4.1 Constructing program embedding space

We follow the approach and the program dataset specified in Trivedi et al. [74] to learn a program embedding space that smoothly and continuously parameterizes programs with diverse behaviors. The training objectives include a VAE loss and two losses that encourage learning a behaviorally smooth program embedding space. Once trained, we can use the learned decoder p θ to map any program embedding z to a program ρ z = p θ ( z ) consisting of a sequence of program tokens. Details on the program dataset generation and the encoder-decoder training can be found in Section E.1.1.

4.2 Retrieving programmatic options

With a program embedding space, we aim to retrieve a set of programs (programmatic options) given a task. This set of programs should have the following properties.

(a) Retrieving Programmatic Options

(b) Learning High-Level Policy with Programmatic Options

def run():

while(markPresent()):

PickMarker()

turnRight()

move()

def run():

if markerPresent():

pickMarker()

else:

move()

def run():

if frontIsClear():

move()

else:

turnLeft()

Decoder

p

θ

Latent

Program Candidates

Z

+

Program

Search Center

z

r

Sample

Decode Each Candidate Program Embedding

z

def run():

while(markPresent()):

PickMarker()

turnRight()

move()

ρ

m

1

ρ

mk

ρ

mk

+1

def run():

if markerPresent():

pickMarker()

else:

move()

def run():

if frontIsClear():

move()

else:

turnLeft()

Retrieved

Programmatic

Options

Environment

Noise

Sampled Program Sequence

Ψ

i

diversity

=

Sigmoid

( - max

z

i

∈

Zk

z

⋅

z

i

∥

z

∥∥

z

i

∥

)

R

Ψ

i

=

1

|

Ψ

i

|

|

Ψ

i

|

∑

j

=1

T j

∑

t

=0

γ

t

/u1D53C

(

s

t

,

a

t

)∼

EXEC

(Ψ

i

[

j

])

[

r

t

]

×

Evaluate Each Candidate Program

High-level

Policy

f

ϕ

Environment

s

i

+1

r

i

+1

s

i

t

a i

t

[

s

i

1

,

. . . ,

s

i

Ti

]

[

r

i

1

,

. . . ,

r

i

Ti

]

[-1]

⋅

Σ

def run():

if markerPresent():

pickMarker()

else:

move()

def run():

while(markPresent()):

PickMarker()

turnRight()

move()

def run():

if markerPresent():

pickMarker()

else:

move()

def run():

if frontIsClear():

move()

else:

turnLeft()

Current Option

m

current

Sample Next

Programmatic Option

ρ

m

next

Execute

ρ

m

next

ρ

m

1

ρ

m

|

M

|

-1

ρ

m

2

m

term

Termination Option

Learning High-Level Policy

f

ϕ

Effectiveness : Each program can solve the task to some extent.

Diversity : The more behaviorally diverse the programs are, the richer behaviors can be captured.

Compatibility : Sequentially executing some programs with specific orders can potentially lead to improved task performance.

4.2.1 Retrieving effective programs

To obtain a task-solving program, we can apply the Cross-Entropy Method [59], iteratively searching in a learned program embedding space [74] in the following procedure:

Randomly initialize a program embedding vector z r as the search center.

Add random noises to z r to generate a population of program embeddings Z = { z i } i =1 , ... ,n , where n denotes the population size.

Evaluate every program embedding z ∈ Z with the evaluation function G to get a list of fitness score [ G ( z i )] i =1 , ... ,n .

Average the top k program embeddings in Z according to fitness scores [ G ( z i )] i =1 , ... ,n and assign it to the search center z r .

Repeat (2) to (4) until the fitness score G ( z r ) of z r converges or the maximum number of steps is reached.

Since we aim to retrieve a set of effective programs, we can define the evaluation function as the program execution return of a decoded program embedding, i . e ., G ( z ) = ∑ T t =0 γ t E ( s t ,a t ) ∼ EXEC ( ρ z ) [ r t ] . To retrieve a set of | M | programmatic options for a high-level policy, we deploy this CEM search N times, take | M | best program embeddings, and obtain the decoded program set { ρ z r i = p θ ( z r i ) } i =1 , ... , | M | . Please refer to Section A.1 for more details and the CEM search pseudo-code.

4.2.2 Retrieving effective, diverse programs

In our proposal, we retrieve a set of programmatic options for a high-level policy. Hence, the diversity of behaviors of the retreived program set endow HIPO with versatility. However, by

running the CEM search for | M | times, the obtained program set can have low diversity, making the multiskill-demanding tasks unsolvable.

To address this issue, we propose the diversity multiplier that accounts previous search results to encourage diversity among the retrieved programs. The evaluation of program employing the diversity multiplier is illustrated in Figure 2. Specifically, during the ( k +1) st CEM search, each program embedding z is evaluated by G ( z, Z k ) = ( ∑ T t =0 γ t E ( s t ,a t ) ∼ EXEC ( ρ z ) [ r t ]) · diversity ( z, Z k ) , where diversity ( z, Z k ) is the proposed diversity multiplier defined as Sigmoid ( -max z i ∈ Z k z · z i ∥ z ∥∥ z i ∥ ) . Thus, the program execution return is scaled down by diversity ( z, Z k ) based on the maximum cosine similarity between z and the retrieved program embeddings Z k = { z i } i =1 , ... ,k from the previous k CEM searches, diverging the current program embedding from previously retrieved programs.

To retrieve a set of | M | programmatic options for our high-level policy, we deploy this CEM+diversity search N times, take | M | best program embeddings, and obtain the decoded program set. The procedure and the search trajectory visualization can be found in Section A.2.

4.2.3 Retrieving effective, diverse, compatible programs

The proposed HIPO executes a sequence of programmatic options determined by a high-level policy. Therefore, these programs shall be compatible with each other, i . e ., executing a program following the execution of other programs could improve task performance. Yet, CEM+diversity discussed in Section 4.2.2 searches every program independently.

Accounting for the compatibility among programs while searching, we propose an evaluation method, CEM+diversity+compatibility. To evaluate the program embedding z , we take the decoded program ρ z as the ( k +1) st option. Then, lists of programs Ψ i,i =1 , ... ,D are sampled with replacements from determined k options and the ( k +1) st option. Each program list Ψ i contains at least one ( k +1) st option to consider the compatibility between the ( k + 1) st and previously determined k options. The return is computed by sequentially executing these D sequences of programs and multiply the result with the diversity multiplier proposed in Section 4.2.2. As the result, the evaluation function is G ( z, Z k ) = 1 D ∑ D i =1 R Ψ i · diversity ( z, Z k ) , where R Ψ i is the normalized reward obtained from executing all programs in the specified program sequence Ψ i :

with | Ψ i | being the number of programs in the program sequence Ψ i , Ψ i [ j ] being the j -th program, and γ as the discount factor.

The search method with the specified evaluation is deployed | M | times to obtain a set of programs that are effective, diverse, and compatible with each other, adopted by the high-level policy as programmatic options. Please refer to Section A.3 for more details and the thorough procedure.

4.3 Learning high-level policy with programmatic options

Given a set of programmatic options M = { m k } k =1 , ..., | M | , we formulate learning a high-level policy f ϕ represented by neural networks, as a reinforcement learning problem aiming to maximize the task return. At the i -th high-level step, given the latest selected programmatic option m i and the current environment state s , the high-level policy f ϕ ( m i , s ) outputs the probability distribution of programmatic options for the next option m i +1 ∈ m term ∪ { m k } k =1 , ..., | M | , where m term denotes the termination option that ends the episode once selected. Otherwise, the corresponding program ρ m i +1 is executed, yielding the next state s i +1 T i +1 and the cumulative reward r i +1 = ∑ T i +1 t =1 r i +1 t . Note that the last state s i +1 T i +1 of the state sequence is returned by EXEC ( ρ m i +1 ) , with T i +1 denoting the horizon of the i +1 -th program execution. Also, the cumulative reward r i +1 obtained within a single program execution is not discounted.

Note that a single program execution EXEC ( ρ ) will terminate after complete execution or the number of function calls emitted during EXEC ( ρ ) reaches 220, which aligns to the setting in Trivedi et al.

(a) SEESAW

(b) UP-N-DOWN

(c) FARMER

(d) INF-DOORKEY

(e) INF-HARVESTER

[74]. This iterative process stops once the termination option is sampled or the maximum number of the option-selection steps is reached. Please refer to Section E.1.3 for training details.

To further enhance the explainability of the high-level policy, we apply the approach proposed by Koul et al. [36] to extract the state machine structure from the learned high-level policy. Combining the retrieved set of programmatic options and the extracted state machine structure, our framework is capable of solving long-horizon tasks while being self-explanatory. Examples of extracted state machines are illustrated in Section D.

5 Experiments

We aim to answer the following questions with the experiments and ablation studies. (1) Can our proposed diversity multiplier introduced in Section 4.2.2 enhance CEM and yield programs with improved performance? (2) Can our proposed CEM+diversity+compatibility introduced in Section 4.2.3 retrieve a set of programs that are diverse yet compatible with each other? (3) Can the proposed framework outperforms existing methods on long-horizon tasks?

Please refer to Section E for hyperparameter settings for the following experiments.

5.1 Karel problem sets

To this end, we consider the Karel domain [56], which is widely adopted in program synthesis [9, 13, 65, 70] and programmatic reinforcement learning [46, 74]. Specifically, we utilize the KAREL problem set [74] and the KAREL-HARD problem set [46]. The KAREL problem set includes six basic tasks, each of which can be solved by a short program (less than 45 tokens), with a horizon shorter than 200 steps per episode. On the other hand, the four tasks introduced in the KAREL-HARD problem require longer, more complex programs ( i . e ., 45 to 120 tokens) in longer execution horizons ( i . e ., up to 500 actions). Details on two problem sets is provided in Section F and Section G.

KAREL-LONG problem set. Since most of the tasks in the KAREL and KAREL-HARD problem sets are short-horizon tasks ( i . e ., can be finished in less than 500 timesteps), they are not suitable for evaluating long-horizon task-solving ability ( i . e ., tasks requiring more than 3000 timesteps to finish). Hence, we introduce a newly designed KAREL-LONG problem set as the benchmark to evaluate the capability of HIPO.

As illustrated in Figure 3, the tasks require the agent to fulfill extra constraints ( e . g ., not placing multiple markers on the same spot in FARMER, receiving penalties imposed for not moving along the stairs in UP-N-DOWN) and conduct extended exploration ( e . g ., repetitively locating and collecting markers in SEESAW, INF-DOORKEY, and INF-HARVESTER). More details on the KAREL-LONG tasks can be found in Section H.

5.2 Cross-entropy method with diversity multiplier

We aim to investigate whether our proposed diversity multiplier can enhance CEM and yield programs with improved performance. To this end, for each KAREL or KAREL-HARD task, we use CEM and

CEM+diversity to find 10 programs. Then, for each task, we evaluate all the programs and report the best performance in Table 1. The results suggest that our proposed CEM+diversity achieves better performance on most of the tasks, highlighting the improved search quality induced by covering wider regions in the search space with the diversity multiplier . Visualized search trajectories of CEM+diversity can be found in Section A.2.

5.3 Ablation study

We propose CEM+diversity+compatibility to retrieve a set of effective, diverse, compatible programmatic options for our high-level policy. In this section, we compare a variety of implementations regarding the diversity and the compatibility of programs.

CEM ×| M | : Conduct the CEM search described in Section 4.2.1 | M | times and take the resulting | M | programs as the set of programmatic options.

CEM+diversity top k , k = | M | : Conduct the CEM search with the diversity multiplier described in Section 4.2.2 N = 10 times and take the top | M | results as the set of programmatic options.

CEM+diversity ×| M | : Conduct the CEM search with the diversity multiplier described in Section 4.2.2 N = 10 times and select the best program as the i th option. Repeat this process | M | times to extract | M | programs as the set of programmatic options.

CEM+compatibility ×| M | : Conduct the CEM search by executing programs in the specified program sequence Ψ i described in Section 4.2.3, excluding the diversity multiplier . Iteratively perform this search | M | times and take the resulting | M | programs as the set of programmatic options.

HIPO (ours) : Conduct CEM+diversity+compatibility ( i . e ., CEM with the diversity multiplier and R Ψ as described in Section 4.2.3) for N = 10 times and select the best result as the i th option. Repeat the above process | M | times and take all | M | results as the set of programmatic options.

The number of programmatic options | M | is 3 for SEESAW, UP-N-DOWN, and INF-HARVESTER and 5 for FARMER and INF-DOORKEY. We assess the quality of the retrieved programmatic options by evaluating the performance of the high-level policy learned with these option sets on the KAREL-LONG tasks. The results presented in Table 2 indicate that our proposed framework, HIPO, outperforms its variants that ignore diversity or compatibility among programmatic options on all the tasks. This justifies our proposed CEM+diversity+compatibility method for retrieving a set of effective, diverse, compatible programs as options for the high-level policy.

5.4 Comparing with deep RL and programmatic RL Methods

We compare our proposed framework and its variant to state-of-the-art deep RL and programmatic RL methods on the KAREL-LONG tasks.

Random transition uses the same set of programmatic options as our method but with a random high-level policy ( i . e ., uniformly randomly select the next option at each step). The performance of this method examines the necessity of learning a high-level policy.

10 1

10 2

10 3

10 4

10 5

10 6

10 7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Farmer

HPRL

LEAPS

Option Searching

HIPO

HC

10 1

10 2

10 3

10 4

10 5

10 6

10 7

0.0

0.2

0.4

0.6

0.8

Inf-Harvester

HPRL

LEAPS

Option Searching

HIPO

HC

# Excecuted Programs

Return

×1

×2

×4

×8

×16

-100

-80

-60

-40

-20

0

Farmer

HIPO

PAO

DRL

LEAPS

HPRL

HC

×1

×2

×4

×8

×16

-80

-60

-40

-20

0

Inf-Harvester

HIPO

PAO

DRL

LEAPS

HPRL

HC

Expected Horizon Length

(a) Program Sample Efficiency

(b) Inductive Generalization

Primitive actions as options (PAO) learns a high-level policy similar to HIPO, which takes the current option and environment state as input and predicts the next option. However, it utilizes primitive actions ( e . g ., move , pickMarker ) as options. This baseline highlights the necessity of retrieving programs with higher-level behaviors as options.

DRL represents a policy as a neural network and is learned using PPO [61]. The policy takes raw states ( i . e ., Karel grids) as input and predicts the probability distribution over the set of primitive actions, ( e . g ., move , pickMarker ).

Option-Critic represents a policy that both high-level and low-level policies are neural networks and is learned using the option-critic architecture [4]. The policy takes raw states ( i . e ., Karel grids) as input, and each option predicts the probability distribution over the set of primitive actions ( e . g ., move , pickMarker ).

Learning Embeddings for Latent Program Synthesis (LEAPS) [74] searches for a single task-solving program using the vanilla CEM in a learned program embedding space.

Hierarchical Programmatic Reinforcement Learning (HPRL) [46] learns a meta-policy, whose action space is a learned program embedding space, to compose a series of programs as the policy.

Hill Climbing (HC) [10] is a stochastic search technique that operates directly within the program space. The process begins by randomly modifying portions of the current program to generate a set of neighboring candidates. The program that performs best among these neighbors is selected as the next candidate for exploration.

As Table 1 shows, HIPO outperforms LEAPS and HPRL on 8 out of 10 tasks from the KAREL and KAREL-HARD tasks, indicating that the retrieved programs are truly effective at solving short horizon

tasks ( i . e ., less than 500 actions). For long-horizon tasks that require more than 3000 actions to solve, Table 2 shows that HIPO excels on four tasks, with better performance on FARMER and particular prowess in SEESAW, UP-N-DOWN, and INF-DOORKEY.

Two of these tasks require distinct skills ( e . g ., pick and put markers in FARMER; go up and downstairs in UP-N-DOWN) and the capability to persistently execute one skill for an extended period before transitioning to another. HIPO adeptly addresses this challenge due to the consideration of diversity while seeking programmatic options, ensuring the acquisition of both skills concurrently.

Unlike the other tasks, SEESAW and INF-DOORKEY require an extended traverse to collect markers, leading to a sparser reward distribution. During the searching phase of programmatic options, emphasizing compatibility enables HIPO to secure a set of mutually compatible options that work together effectively to accomplish the extended traversal.

Retrieved programs are provided in Appendix (Figure 22, Figure 23, Figure 24, and Figure 25). Experimental results on programmatic policy baselines over 32 seeds are provided in Table 3.

5.5 Program sample efficiency

To accurately evaluate the sample efficiency of programmatic RL methods, we propose the program sample efficiency metric, measuring the total number of program executions required to learn a program policy. We report the program sample efficiency of LEAPS, HPRL, HC, and HIPO on FARMER and INF-HARVESTER in Figure 4a. As the results show, HIPO demonstrates program sample efficiency than LEAPS and HPRL, indicating that our framework requires fewer program interactions with the environment and lower computational costs compared to existing latent-spacebased programmatic RL frameworks. More details and the action sample efficiency can be found in Section B and Figure 8.

5.6 Inductive generalization

We aim to compare the inductive generalization ability among all the methods, generalizing to out-of-distributionally ( i . e ., unseen during training) long task instances [32]. To this end, we increase the expected horizons of FARMER and INF-HARVESTER by 2 × , 4 × , 8 × , and 16 × . Then, we report the performance drop compared to the original task performance of selected baselines in Figure 4b. More details on extending task horizons are provided in Section C.

The results show that HIPO suffers a fewer decline in performance in testing environments with significantly extended horizons than LEAPS and HPRL, suggesting that HIPO exhibits better inductive generalization in these tasks. The longest execution of HIPO ran up to 48 k environment steps.

5.7 Interpretability

In the proposed framework, we retrieve a set of programmatic options that can be reused by the highlevel policy to solve long and repetitive tasks. For example, in INF-DOORKEY (detailed in Section H), the retrieved programmatic options are presented in Figure 24. Based on these programmatic options, the high-level policy can reuse them to guide the agent to traverse all four chambers with the following sequence of programmatic options: τ m = { m 5 , m 3 , m 5 , m 3 , m 1 , m 4 , m 4 , m 2 , m 5 , m 4 , m 5 } .

INF-DOORKEY requires three different skills to solve: picking a marker to open the door in some chambers, placing a marker to open the door in some other chambers, and navigating between the chambers. Specifically, the programs provided in Figure 24 show that Option 1 and Option 5 fulfill the first skill, while Option 3, Option 4, and Option 5 meet the requirements for the second, and Option 2 satisfy the third. Along with the observation, we could fully interpret the agent policy with the control flows, perceptions, and actions specified in the program. On the other hand, the high-level policy is a neural network by itself, making it hard to interpret the option transition dynamic learned by the high-level policy.

To improve the interpretability of our high-level policy, we extract state machine structures from the high-level policy to visualize the option transition dynamics. More details and examples of extracted state machines can be found in Section D.

6 Conclusion

This work aims to construct reinforcement learning policies that are human-interpretable and generalizable, bridging from hierarchical reinforcement learning to programmatic options. Consequently, we propose the Hierarchical Programmatic Option framework (HIPO) to represent complex behaviors and address long-horizon tasks. Specifically, we introduce a method that can retrieve a set of effective, diverse, compatible programs by modifying the Cross Entropy Method (CEM). Following this, these programs are applied as options by the high-level policy learned with reinforcement learning. To evaluate HIPO's ability in extended horizons, we design a set of tasks that require thousands of steps in the Karel domain. Our framework HIPO outperforms various deep RL and programmatic RL methods on various tasks. In addition, HIPO demonstrates good performance in inductive generalization to even longer horizons without fine-tuning. Last but not least, extensive ablation studies justify the effectiveness of our proposed search algorithm in retrieving programmatic options.

References

David S Aleixo and Levi HS Lelis. Show me the way! bilevel search for synthesizing programmatic strategies. In Association for the Advancement of Artificial Intelligence , 2023.

Mihai Anca, Mark F. Hansen, and Matthew Studley. Modular hierarchical reinforcement learning for robotics: Improving scalability and generalizability. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems , 2023.

Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning , 2017.

Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Association for the Advancement of Artificial Intelligence , 2017.

Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. In International Conference on Learning Representations , 2017.

Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems , 2003.

Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. In Neural Information Processing Systems , 2018.

Benjamin Beyret, Ali Shafti, and A. Aldo Faisal. Dot-to-dot: Explainable hierarchical reinforcement learning for robotic manipulation. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5014-5019, 2019.

Rudy R Bunel, Matthew Hausknecht, Jacob Devlin, Rishabh Singh, and Pushmeet Kohli. Leveraging grammar and reinforcement learning for neural program synthesis. In International Conference on Learning Representations , 2018.

Tales Henrique Carvalho, Kenneth Tjhia, and Levi Lelis. Reclaiming the source of programmatic policies: Programmatic versus latent spaces. In The Twelfth International Conference on Learning Representations , 2024.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex

Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Shang-Fu Chen, Hsiang-Chun Wang, Ming-Hao Hsu, Chun-Mao Lai, and Shao-Hua Sun. Diffusion model-augmented behavioral cloning. In International Conference on Machine Learning , 2024.

Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In International Conference on Learning Representations , 2019.

Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. arXiv preprint arXiv:2306.13831 , 2023.

Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoderdecoder for statistical machine translation. arXiv , 2014.

Dongkyu Choi and Pat Langley. Learning teleoreactive logic programs from problem solving. In International Conference on Inductive Logic Programming , 2005.

Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning , 2019.

Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. In Neil D. Lawrence and Mark Girolami, editors, Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics , volume 22 of Proceedings of Machine Learning Research , pages 273-281, La Palma, Canary Islands, 21-23 Apr 2012. PMLR.

Quentin Delfosse, Hikaru Shindo, Devendra Dhami, and Kristian Kersting. Interpretable and explainable logical policies via neurally guided symbolic abstraction. arXiv , 2023.

Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. In International Conference on Machine Learning , 2017.

Fan Ding and Fei Zhu. Hliferl: A hierarchical lifelong reinforcement learning framework. Journal of King Saud University - Computer and Information Sciences , 34(7):4312-4321, 2022. ISSN 1319-1578.

Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv preprint arXiv:2006.08381 , 2020.

Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In Competition and Cooperation in Neural Nets: Proceedings of the US-Japan Joint Seminar , 1982.

Daniel Graves, Jun Jin, and Jun Luo. Lispr: An options framework for policy reuse with reinforcement learning. arXiv , 2020.

Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv , 2016.

Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In IEEE International Conference on Robotics and Automation , 2017.

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations , 2016.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems , 2016.

Joey Hong, David Dohan, Rishabh Singh, Charles Sutton, and Manzil Zaheer. Latent programmer: Discrete latent codes for program synthesis. In International Conference on Machine Learning , 2021.

Bo-Ruei Huang, Chun-Kai Yang, Chun-Mao Lai, , and Shao-Hua Sun. Diffusion imitation from observation. In Neural Information Processing Systems , 2024.

Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research , 2021.

Jeevana Priya Inala, Osbert Bastani, Zenna Tavares, and Armando Solar-Lezama. Synthesizing programmatic policies that inductively generalize. In International Conference on Learning Representations , 2020.

Zhengyao Jiang and Shan Luo. Neural logic reinforcement learning. arXiv , 2019.

Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Ja´ skowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In IEEE Conference on Computational Intelligence and Games , 2016.

Hector Kohler, Quentin Delfosse, Riad Akrour, Kristian Kersting, and Philippe Preux. Interpretable and editable programmatic tree policies for reinforcement learning. arXiv preprint arXiv:2405.14956 , 2024.

Anurag Koul, Alan Fern, and Sam Greydanus. Learning finite state representations of recurrent policy networks. In International Conference on Learning Representations , 2019.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 2017.

Chun-Mao Lai, Hsiang-Chun Wang, Ping-Chun Hsieh, Yu-Chiang Frank Wang, Min-Hung Chen, and Shao-Hua Sun. Diffusion-reward adversarial imitation learning. In Neural Information Processing Systems , 2024.

Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio P Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep reinforcement learning. In International Conference on Machine Learning , 2021.

Youngwoon Lee, Shao-Hua Sun, Sriram Somasundaram, Edward Hu, and Joseph J. Lim. Composing complex skills by learning transition policies. In International Conference on Learning Representations , 2019.

Youngwoon Lee, Andrew Szot, Shao-Hua Sun, and Joseph J. Lim. Generalizable imitation learning from observation via inferring goal proximity. In Neural Information Processing Systems , 2021.

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science , 2022.

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In International Conference on Robotics and Automation , 2023.

Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D Ernst. Nl2bash: A corpus and semantic parser for natural language interface to the linux operating system. In International Conference on Language Resources and Evaluation , 2018.

Zachary C Lipton. The mythos of model interpretability. In ICML Workshop on Human Interpretability in Machine Learning , 2016.

Guan-Ting Liu, En-Pei Hu, Pu-Jen Cheng, Hung-Yi Lee, and Shao-Hua Sun. Hierarchical programmatic reinforcement learning via learning to compose programs. In International Conference on Machine Learning , 2023.

Max Liu, Chan-Hung Yu, Wei-Hsu Lee, Cheng-Wei Hung, Yen-Chun Chen, and Shao-Hua Sun. Synthesizing programmatic reinforcement learning policies with large language model guided search. arXiv preprint arXiv:2405.16450 , 2024.

Yunchao Liu, Jiajun Wu, Zheng Wu, Daniel Ritchie, William T. Freeman, and Joshua B. Tenenbaum. Learning to describe scenes with programs. In International Conference on Learning Representations , 2019.

Lirui Luo, Guoxi Zhang, Hongming Xu, Yaodong Yang, Cong Fang, and Qing Li. End-to-end neuro-symbolic reinforcement learning with textual explanations. arXiv , 2024.

Zhihao Ma, Yuzheng Zhuang, Paul Weng, Hankz Hankui Zhuo, Dong Li, Wulong Liu, and Jianye Hao. Learning symbolic rules for interpretable deep reinforcement learning. arXiv , 2021.

Timothy Mann and Shie Mannor. Scaling up approximate value iteration with options: Better policies with fewer iterations. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research , pages 127-135, Bejing, China, 22-24 Jun 2014. PMLR.

Leandro C Medeiros, David S Aleixo, and Levi HS Lelis. What can we learn even from the weakest? learning sketches for programmatic strategies. In Association for the Advancement of Artificial Intelligence , 2022.

Rubens O Moraes and Levi HS Lelis. Searching for programmatic policies in semantic spaces. arXiv preprint arXiv:2405.05431 , 2024.

Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, and Joseph J. Lim. Skill-based meta-reinforcement learning. In International Conference on Learning Representations , 2022.

Kaleb Ben Naveed, Zhiqian Qiao, and John M. Dolan. Trajectory planning for autonomous vehicles using hierarchical reinforcement learning. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC) , pages 601-606, 2021.

Richard E Pattis. Karel the robot: a gentle introduction to the art of programming . John Wiley &Sons, Inc., 1981.

Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning , 2021.

Zhiqian Qiao, Zachariah Tyree, Priyantha Mudalige, Jeff Schneider, and John M. Dolan. Hierarchical reinforcement learning method for autonomous vehicle behavior planning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 6084-6089, 2020.

Reuven Y Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research , 1997.

Stefan Schaal. Learning from demonstration. In Advances in Neural Information Processing Systems , 1997.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 8624-8633. PMLR, 13-18 Jul 2020.

Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamicsaware unsupervised discovery of skills. In International Conference on Learning Representations , 2020.

Owen Shen. Interpretability in ML: A broad overview, 2020.

Eui Chul Shin, Illia Polosukhin, and Dawn Song. Improving neural program synthesis with inferred execution traces. In Neural Information Processing Systems , 2018.

David Silver and Kamil Ciosek. Compositional planning using optimal option models. In International Conference on Machine Learning , 2012.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature , 2016.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature , 2017.

Tom Silver, Kelsey R Allen, Alex K Lew, Leslie Pack Kaelbling, and Josh Tenenbaum. Few-shot bayesian imitation learning with logical program policies. In Association for the Advancement of Artificial Intelligence , 2020.

Shao-Hua Sun, Hyeonwoo Noh, Sriram Somasundaram, and Joseph Lim. Neural program synthesis from diverse demonstration videos. In International Conference on Machine Learning , 2018.

Shao-Hua Sun, Te-Lin Wu, and Joseph J. Lim. Program guided agent. In International Conference on Learning Representations , 2020.

Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence , 1999.

Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Learning to infer and execute 3d shape programs. In International Conference on Learning Representations , 2019.

Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph J Lim. Learning to synthesize programs as interpretable and generalizable policies. In Neural Information Processing Systems , 2021.

Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning , 2018.

Abhinav Verma, Hoang Le, Yisong Yue, and Swarat Chaudhuri. Imitation-projected programmatic reinforcement learning. In Neural Information Processing Systems , 2019.

Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning , 2017.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 2019.

Huaxiaoyue Wang, Gonzalo Gonzalez-Pumariega, Yash Sharma, and Sanjiban Choudhury. Demo2code: From summarizing demonstrations to synthesizing code via extended chain-ofthought. arXiv preprint arXiv:2305.16744 , 2023.

Elly Winner and Manuela Veloso. Distill: Learning domain-specific planners by example. In International Conference on Machine Learning , 2003.

Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In IEEE Conference on Computer Vision and Pattern Recognition , 2017.

Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature , 2022.

Duo Xu and Faramarz Fekri. Interpretable model-based hierarchical reinforcement learning using inductive logic programming. arXiv , 2021.

Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893 , 2018.

Linghan Zhong, Ryan Lindeborg, Jesse Zhang, Joseph J Lim, and Shao-Hua Sun. Hierarchical neural program synthesis. arXiv preprint arXiv:2303.06018 , 2023.

Appendix

Table of Contents

A Cross entropy method details

A.1 CEM

Figure 5 illustrates the workflow of the cross entropy method (CEM). The corresponding pseudo-code is available in Algorithm 1.

Hyperparameters list:

Population size n : 64

Standard Deviation of Noise σ : 0.5

Percent of the Population Elites e : 0.05

Exponential σ decay: True

Maximum Iteration N s : 1000

A.2 CEM+diversity

The procedure of running CEM+diversity N times is as follows:

Search the 1 st program embedding z 1 by CEM ( G,g = ( Z k : {} ))

Search the 2 nd program embedding z 2 by CEM ( G,g = ( Z k : { z 1 } ))

...

(N) Search the Nth program embedding z N by CEM ( G,g = ( Z k : { z 1 , ..., z N -1 } ))

Z k is the set of retrieved program embeddings { z i } i =1 , ... ,k -1 from the previous ( k -1) CEMsearches. The evaluation function is G ( z, Z k ) = ( ∑ T t =0 γ t E ( s t ,a t ) ∼ EXEC ( ρ z ) [ r t ]) · diversity ( z, Z k ) , where diversity ( z, Z k ) = Sigmoid ( -max z i ∈ Z k z · z i ∥ z ∥∥ z i ∥ ) . Searching trajectories shown in Figure 6 exemplifies the influence of the diversity factor.

...

Noise

Sampling

TopK

Candidate

Combination

Program

Evaluation

Function

Program

Embedding

after

iteration

For

Each

Algorithm 1 Cross Entropy Method

1: Input: Evaluation Function G , Function Input g , Maximum Iteration N s , Population Size n , Standard Deviation of Noise σ , Percent of the Population Elites e .

2: Latent Program Search Center z r ← [ z 0 , z 1 , ..., z i , ..., z 255 ] , z i ∼ N (0 , 1) 3: step ← 0 4: while step < N s do 5: Candidate Latent Programs Z ← [ ] 6: Fitness Scores L G ← [ ] 7: for i ← 1 to n do 8: ε ← [ ε 0 , ε 1 , ..., ε i , ..., ε 255 ] , ε i ∼ N (0 , σ ) 9: Z .append ( z r + ε ) 10: L G .append ( G (( z r + ε ) , g )) 11: end for 12: Elite Latent Programs Z kl ← Latent Programs in top e percent of Z ranked by L G . 13: z r ← mean ( Z kl ) 14: step ← step +1

15: end while

CEM0

30

Z_0

CEM 1

Z_1

20

CEM2

z_2

10

0

-10

-20

-20

-10

0

10

20

30

A.3 CEM+diversity+compatibility

A.3.1 Sample program sequence

In the subsequent section, we outline the procedure for sampling a program sequence Ψ from k previously determined programs ρ i,i =1 , ... ,k during the search of the ( k +1) st program ρ k +1 .

Uniformly sample a program ρ j from all k +1 programs { ρ 1 , ... , ρ k , ρ k +1 } , and add ρ j to Ψ .

Repeat (1) until the ( k +1) st program ρ k +1 is sampled.

Uniformly sample a program ρ j from { ρ 1 , ... , ρ k , ρ k +1 , ρ term } , where ρ term represents the termination program, then append ρ j to Ψ .

Repeat (3) until ρ term is sampled.

Return Ψ with length greater or equal to L min , where L min is a hyperparameter that indicates the minimum length of a sampled sequence. Re-sample Ψ otherwise.

A.3.2 The score function

The score function G ( z, Z k ) for CEM+diversity+compatibility:

| Ψ i | denotes the number of programs in the program list Ψ i , Ψ i [ j ] represents the j -th program in the program list Ψ i , and γ is the discount factor.

A.3.3 CEM+diversity+compatibility procedure

The procedure of running CEM+diversity+compatibility | M | times in order to retrieve | M | programs:

Retrieve 1 st program z 1 .

Sample a program sequence Ψ i =1 with k = 0

Deploy CEM+diversity N times with Z k = {} to acquire N program embeddings.

Select the program embedding with the highest score G ( z, Z k = {} ) as z 1 , among the N program embeddings.

Retrieve 2 nd program z 2 .

Sample a program sequence Ψ i =1 , 2 with k = 1 previously determined program.

Deploy CEM+diversity N times with Z k = { z 1 } , to acquire N program embeddings.

Select the program embedding with the highest score G ( z, Z k = { z 1 } ) as z 2 , among the N program embeddings.

...

( | M | ) Retrieve | M | th program z | M | .

Sample a program sequence Ψ i,i =1 ,..., 2 | M |-1 with k = | M | -1 previously determined programs.

Deploy CEM+diversity N times with Z k = { z 1 , z 2 , ..., z | M |-1 } , to acquire N program embeddings.

Select the program embedding with the highest score G ( z, Z k = { z 1 , z 2 , ..., z | M |-1 } ) as z | M | , among the N program embeddings.

10 1

10 2

10 3

10 4

10 5

10 6

10 7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Farmer

HPRL

LEAPS

Option Searching

HIPO

HC

10 1

10 2

10 3

10 4

10 5

10 6

10 7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Inf-DoorKey

HPRL

LEAPS

Option Searching

HIPO

HC

10 1

10 2

10 3

10 4

10 5

10 6

10 7

0.0

0.2

0.4

0.6

0.8

Inf-Harvester

HPRL

LEAPS

Option Searching

HIPO

HC

# Excecuted Programs

Return

10 2

10 4

10 6

10 8

10 10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Farmer

HPRL

LEAPS

Option Searching

HIPO

HC

10 2

10 4

10 6

10 8

10 10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Inf-DoorKey

HPRL

LEAPS

Option Searching

HIPO

HC

10 2

10 4

10 6

10 8

10 10

0.0

0.2

0.4

0.6

0.8

Inf-Harvester

HPRL

LEAPS

Option Searching

HIPO

HC

# Excecuted Actions

Return

B Program sample efficiency

Generally speaking, programmatic RL approaches incorporate a three-step training procedure, in which a program will be first synthesized, exectued, then evaluated with respect to a given task. The three-step procedure is iteratively applied until the return converges or the maximum training step is reached. "Program Sample Efficiency"-referring to how effectively the method achieves a desired return using an efficient number of the three-step procedure-is crucial for analyzing and comparing the efficiency of our approaches against the baselines. As Figure 7 illustrates, HIPO (our method) obtains the best program sampling efficiency in FARMER and INF-DOORKEY. Details of the program sample efficiency calculation for each baseline are elaborated in the subsequent sections.

B.1 HIPO

In the program searching phase of HIPO, up to 50 CEM-searches are conducted to fetch programmatic options. Each CEM-search involves a maximum of 1000 search iterations, in which the three-step procedure (synthesis, execution, and evaluation) is executed n times, where n is the population size of the CEM-search. In the first phase of the efficiency curve of our method (indicated by the yellow curve in Figure 7), the return is counted as the highest return obtained from a sequential execution of the programmatic options, based on the order indicated by the sampled random sequence.

Afterward, during the high-level policy training phase (indicated by the red curve in Figure 7), the three-step procedure is executed once per PPO training step. The return is recorded as the maximum validation return achieved in the HIPO manner, where the high-level policy continuously selects and deploys a program to solve a long-horizon task.

B.2 LEAPS

In the program searching process of LEAPS, 216 CEM-searches are performed to obtain the targeted program, in accordance with the settings in [74]. In each CEM-search, up to 1000 iterations are performed, in which the three-step procedure is executed n times, where n is the population size of

×1

×2

×4

×8

×16

-100

-80

-60

-40

-20

0

Farmer

HIPO

PAO

DRL

LEAPS

HPRL

HC

×1

×2

×4

×8

×16

-100

-80

-60

-40

-20

0

Inf-DoorKey

HIPO

PAO

DRL

LEAPS

HPRL

HC

×1

×2

×4

×8

×16

-80

-60

-40

-20

0

Inf-Harvester

HIPO

PAO

DRL

LEAPS

HPRL

HC

Expected Horizon Length

the CEM-search. The return for a certain number of executed programs is recorded as the maximum return obtained from executing the previously searched programs only.

B.3 HPRL

During the meta-policy training process of HPRL, the three-step procedure is performed once per PPO training step. Therefore, with the exact experiment settings described in Section E.6, the three-step procedure would be executed 25M times at the end of the training process. The return for a certain number of executed programs is recorded as the maximum return achieved in the cascaded execution of 5 programs, decoded from the latent program embeddings output by the meta-policy.

B.4 HC

In the program searching progress of HC, 250 neighbors are sampled and executed to obtain the targeted program for each iteration. Up to 1000000 evaluations would be performed in the searching process, with 16 parallel execution of a program involved in one evaluation. The return for a certain number of executed programs is recorded as the maximum return obtained from executing the previously searched programs only.

C Inductive generalization

To compare baselines with our proposal regarding inductive generalization, the expected horizon of the environment is scaled up by increasing the upper limit of the target for each KAREL-LONG task during the testing phase. For instance, in the FARMER task, the upper limit number is essentially the maximum iteration number of the filling-and-collecting rounds that we expect the agent to accomplish (please refer to Section H for more details). In the original task setting, the agent is tasked to continuously placing and picking markers in a total of 10 rounds of the filling-and-collecting process. In other words, all policies among baselines are trained to terminate after 10 placing-andpicking iterations. Nevertheless, the upper limit number is set to 20, 40, etc, in the testing environment regarding the inductive generalization ability of each policy.

Since most of the baselines obtain a poor peformance on SEESAW and UP-N-DOWN ( i . e ., more than half of baseline approaches have mean return close to 0 . 0 on these tasks), we conduct experiments mainly on FARMER, INF-DOORKEY, and INF-HARVESTER. The testing environments will have expected horizon lengths that are 2, 4, 8, and 16 times longer than those of the training environments. Additionally, the rewards for picking or placing markers, as well as the penalties for actions, will be scaled down by factors of 2, 4, 8, and 16, respectively, ensuring that the maximum accumulated reward for each task is normalized to 1. The detailed settings and experimental results for each of these three tasks are presented below.

C.1 FARMER

During the training phase of all baselines, the maximum iteration number is set to 10. Afterward, we modified this number to 20, 40, 80, and 160 in the testing phase. As shown in Figure 9, as the expected horizon length increases, the performance of all baselines except HIPO declines significantly. This

indicates that our method has a much better inductive generalization property for this task. Further details on the definition of the maximum iteration number are provided in Section H.

C.2 INF-HARVESTER

During the training phase of all baslines, the emerging probability is set to 1 2 . Following this, we modified this number to 3 4 , 7 8 , 15 16 and 31 32 in the testing phase. As shown in Figure 9, as the expected horizon length grows, the performances of HIPO, PAO, and DRL drop slightly, but the performances of LEAPS and HPRL drop extensively. Also, HC performs better when the scale increases, but drops when the scale grows further. More details on the definition of the emerging probability are provided in Section H.

C.3 INF-DOORKEY

During the training phase of all baselines, the upper limit number of marker-picking and markerplacing is set to 16. Then, we modified this number to 32, 64, 128, and 256 in the testing phase. As shown in Figure 9, as the expected horizon length grows, the performance of all the baselines drop significantly. Nevertheless, HIPO has a minor performance drop compared to other baselines. Further details on the definition of the upper limit number are provided in Section H.

D State machine extraction

In our approach, since the high-level neural network policy is incorporated, the proposed HIPO is only partially or locally interpretable - once the the high-level policy selects a programmatic option, human users can recognize the following execution of the program.

To further increase the interpretability of the trained high-level policy f ϕ , we extracted a state machine structure by the approach proposed in [36]. In this setup, since HIPO utilizes the previous programmatic option as input and predicts the following option, we focus only on encoding the environment state observations. Each state observation is processed by convolutional neural networks and fully connected layers into a 1 × 32 vector, which is then quantized into a 1 × h vector. The hyperparameter h balances between the simplicity of the finite state machine and performance drop. Using these quantized vectors, we can construct a state-transition table. The final step involves minimizing these quantized vectors to effectively represent the state machine's structure. Examples of the extracted state machines are shown in Figure 10, Figure 11, and Figure 12.

E Hyperparameters and experimental settings

E.1 HIPO

E.1.1 Encoder & decoder

We follow the training procedure and the model structure proposed in [74], which uses the GRU [15] network to implement both the encoder q ψ and the decoder p θ with hidden dimensions of 256. The encoder q ψ and decoder p θ are trained on programs randomly sampled from the Karel DSL. The

start

M4

M1

O2, O4, O5, O6, O7, O8, O9, O12, O13, O15, O18, O21, O24, O25, O26, O27, O30

M3

O1, O16, O23, O29

O10, O11, O14, O17, O19, O20, O22, O28, O31

end

O3

M2

O2, O4, O5, O7, O8, O12, O13, O17

O1, O6, O9, O11, O14, O15, O16, O20, O21, O23, O24, O25, O26, O28, O30

O10, O18, O19, O22, O27, O29, O31

O3

O4, O9, O12, O13, O30

O1, O2, O5, O6, O7, O8, O11, O14, O15, O16, O17, O18, O20, O21, O22, O23, O24, O25, O29

O10, O19, O26, O27, O28, O31

O3

O2, O4, O5, O13, O25, O26, O30

O8, O9, O15, O16, O18, O20, O24, O27, O29

O1, O6, O7, O10, O11, O12, O14, O17, O19, O21, O22, O23, O28, O31

O3

M5

O4, O5, O7, O11, O12, O18, O21, O23, O24, O26, O30

O1, O2, O6, O8, O9, O10, O13, O14, O15, O16, O20, O27, O28, O29

O17, O19, O22, O25, O31

O3

start

M3

O11

M1

M2

O10

O1, O3, O8, O9, O11

M4

O7

M5

O4, O5, O6

end

O2

O8, O9, O10

O1, O3, O11

O4, O5, O7

O6

O2

O4

O10

O1, O3, O8, O9, O11

O7

O5, O6

O2

O10

O1, O3, O8, O9, O11

O7

O4, O5, O6

O2

O8, O10

O1, O9, O11

O4, O7

O3, O5, O6

O2

start

M1

O2

O1, O2, O3

end

O4

M2

O1, O2, O3, O4

M3

O1, O2, O4

O3

Current

Option

State

Next Option

Execution

Actions

def run():

while frontIsClear():

move()

putMarker()

def run():

while not frontIsClear():

move()

turnLeft()

def run():

while noMarkerPresent():

putMarker()

move()

def run():

while markerPresent():

pickMarker()

move()

Environment

State

Termination

Reward

High-level Policy

loss function for training the encoder-decoder model integrates the β -VAE [27] loss, the program behavior reconstruction loss [74], and the latent behavior reconstruction loss [74].

The program dataset used to train q ψ and p θ consists of 35,000 programs for training and 7,500 programs for validation and testing. Program tokens are sequentially sampled for each program based on defined probabilities until an ending token is reached or the maximum program length of 40 is attained. Defined probabilities of each type of token are listed below:

WHILE : 0.15

REPEAT : 0.03

STMT_STMT : 0.5

ACTION : 0.2

IF : 0.08

IFELSE : 0.04

Note: the token STMT_STMT represents a division operation that splits the current token into two separate tokens, which are then sampled following the above probability list. This token primarily dictates the program's length, as well as the quantity and complexity of nested loops and statements.

E.1.2 Programmatic options

We conduct the procedure described in Section A.3.3 to search programmatic options for each highlevel policy. For tasks only involving skills of marker-picking and traversal, i.e., INF-HARVESTER, SEESAW, and UP-N-DOWN, we designate the number of programmatic options as | M | = 3 , while for the remaining tasks, | M | = 5 .

E.1.3 High-level policy

The high-level policy f ϕ , whose training process is illustrated in Figure 13, comprises convolutional layers [23, 37] to extract features from the Karel states and fully connected layers to determine the next program for execution. Meanwhile, one-hot encodings are used by the high-level policy to represent the indices of programmatic options. The detailed setting of the convolutional layers accords with the description in Section E.3. PPO [61] algorithm is adopted to optimize the high-level policy f ϕ . Hyperparameters are listed below:

Maximum program number: 1000

Batch size : 32

Clipping: 0.05

α : 0.99

γ : 0.99

GAE lambda: 0.95

Value function coefficient: 0.5

Entropy coefficient: 0.1

Number of updates per training iteration: 4

Number of environment steps per set of training iterations: 32

Number of parallel actors: 32

Optimizer : Adam

Learning rate: {0.1, 0.01, 0.001, 0.0001, 0.00001}

E.2 PAO

The implemented PAO resembles the setting in Section E.1. The input, output, and structure of the high-level policy f ϕ remain the same. However, the 5 low-level programs are replaced by 5 primitive actions ( move , turnLeft , turnRight , putMarker , pickMarker ).

E.3 DRL

DRL training on the Karel environment is implemented with the PPO [61] algorithm with 20 million timesteps. Both the policy and value networks share a convolutional encoder interpreting the state of the grid world. This encoder comprises two layers. The first layer comprises 32-filters, a size-4 kernel, and a stride of 1. The following layer has 32 filters, a size-2 kernel, and the same stride of 1. The high-level policy neural network yields the probability distribution among primitive actions (move, turnLeft, turnRight, putMarker, pickMarker) and termination, given the current environmental state as input. In our experiments with DRL on Karel-Long tasks, all hyperparameters are fixed as listed below, except for learning rates tuned in a grid-search manner.

Maximum horizon: 50000

Batch size : 32

Clipping: 0.05

α : 0.99

γ : 0.99

GAE lambda: 0.95

Value function coefficient: 0.5

Entropy coefficient: 0.1

Number of updates per training iteration: 4

Number of environment steps per set of training iterations: 128

Number of parallel actors: 32

Optimizer : Adam

Learning rate: {0.1, 0.01, 0.001, 0.0001, 0.00001}

E.4 Option-Critic

Option-Critic training on the Karel environment is implemented with the option-critic architecture [4]. algorithm with 20 million timesteps. The raw state inputs are extracted by an encoder comprised of two layers. The first layer comprises 32 filters, a size-4 kernel, and a stride of 1. The following layer has 32 filters, a size-2 kernel, and the same stride of 1. In our experiments with Option-Critic on Karel-Long tasks, all hyperparameters are listed below.

Maximum horizon: {2000, 4000, 6000, 8000, 10000}

Number of options: {2, 3, 4, 5}

Batch size : {32, 64, 128} · γ : 0.99 · Termination Regularization: {0.1, 0.01} · Freeze interval: {128, 256, 512, 1024} · Update frequency: {32, 64, 128} · Entropy regularization: {0.1, 0.01}

Learning rate: {0.1, 0.01, 0.001, 0.0001, 0.00001}

E.5 LEAPS

Following the setup detailed in [74], we conducted experiments with various CEM-related hyperparameters as listed below, in order to optimize rewards for LEAPS.

Population size (n): {8, 16, 32, 64} · σ : {0.1, 0.25, 0.5} · e : {0.05, 0.1, 0.2} · Exponential σ decay: {True, False}

Initial distribution P : { N (1 , 0) , N (0 , σ ) , N (0 , 0 . 1 σ ) }

E.6 HPRL

Aligned with the approach described in [46], we trained the meta-policy for each task to predict a program sequence, with hyperparameters listed below.

Max subprogram: 5

Max subprogram Length: 40

Batch size : 128

Clipping: 0.05

α : 0.99

γ : 0.99

GAE lambda: 0.95

Value function coefficient: 0.5

Entropy coefficient: 0.1

Number of updates per training iteration: 4

Number of environment steps per set of training iterations: 32

Number of parallel actors: 32

Optimizer : Adam

Learning rate: 0.00001

Training steps: 25M

E.7 HC

According to [10], we performed the stochastic hill climbing search to find the resulting program. Hyperparameters are listed below.

Number of neighbor candidate programs: 250

Number of parallel actors: 16

Total number of evaluated programs: 1000000

Maximum function calls: 10000

F Details of KAREL problem set

Introduced in Trivedi et al. [74], the KAREL problem set includes 6 tasks: STAIRCLIMBER, FOURCORNER, TOPOFF, MAZE, CLEANHOUSE and HARVESTER. Figure 14 and Figure 15 depict randomly generated initial states, legit sampled internal states, and desired final states for each task. The experimental results in Table 1 average 32 rewards obtained across 32 randomly generated initial states of the environment.

F.1 STAIRCLIMBER

In a 12 × 12 grid environment, the agent's task is to climb the stairs and reach designated marked grid. Both the agent's starting position and the marked grid's position are randomly initialized on the stairs, with the marked grid always placed at the higher end. The reward is sparsely defined as: the agent receives a reward of 1 for reaching the goal, -1 for moving off the stairs, and 0 otherwise.

F.2 FOURCORNER

In a 12 × 12 grid environment, the agent is tasked with placing a marker at each of the four corners. The agent receives no reward for placing a marker anywhere on the grid other than the four corners. Each successful marker placement at an unmarked corner earns a reward of 0 . 25 .

F.3 TOPOFF

In a 12 × 12 grid environment, the agent's objective is to place a marker on every grid cell in columns where the bottom row has a marker in presence. Moreover, the agent should end up in the rightmost square of the row at the end of the episode. The agent receives rewards for consecutively and correctly placing markers until a mistake, which occurs if it places a marker on an empty cell in the bottom row or a marked cell.

F.4 MAZE

In an 8 × 8 grid environment, the agent is tasked to find a marker by navigating the grid environment. The marker's location, the initial agent position, and the maze layout configuration are randomly initialized. The agent receives a sparse reward of 1 for successfully finding the marker in the environment, and 0 otherwise.

F.5 CLEANHOUSE

In a 14 × 22 grid environment, the agent's target is to collect scattered markers as many as possible. The initial location of the agent is fixed, whereas the scattered markers are randomly placed within the environment, adjacent to walls. The return is calculated as the ratio of the number of collected markers to the total number of initially designated markers.

F.6 HARVESTER

In an 8 × 8 grid environment, initially populated with markers in all grid cells, the agent's objective is to pick up a marker from each location within. The return is calculated as ratio of the number of picked markers to the total number of initially placed markers.

G Details of KAREL-HARD problem set

The KAREL-HARD problem set proposed by Liu et al. [46] involves 5 tasks: DOORKEY, ONESTROKE, SEEDER and SNAKE. Each task in this benchmark is designed to be more constrained and structurally complex than those in the KAREL problem set. Figure 21 depicts randomly generated initial states, legit sampled internal states, and final states for each task. The experimental results in Table 1 average 32 rewards obtained across 32 randomly generated initial states of the environment.

(a) STAIRCLIMBER

(b) FOURCORNER

(c) TOPOFF

(d) MAZE

回

回

0

业

回

回

G.1 DOORKEY

In an 8 × 8 grid environment, partitioned into a 6 × 3 left room and a 6 × 2 right room by a column of walls, the agent's objective is to collect a key (represented by a marker) in the left room to unlock a door (an empty grid cell subsequently occurs in that wall column) and place the key on a marked grid cell in the right room. The agent's initial location, the key's location, and the target's location are randomly initialized. The agent receives a reward of 0 . 5 for collecting the key and another 0 . 5 for placing the key on the target.

(a) CLEANHOUSE

向

回

(b) HARVESTER

G.2 ONESTROKE

In an 8 × 8 grid environment, the agent is tasked to navigate through all grid cells without revisiting any of them. An empty grid cell is replaced with a wall once visited. The episode ends once the agent collides with any of these walls. The return is calculated as the ratio of visited grids to the total number of empty grids in the initial environment.

G.3 SEEDER

In an 8 × 8 grid environment, the agent's task is to place one marker on every grid cell. The episode terminates immediately whenever the agent places multiple markers on one grid cell. The return is calculated as the ratio of the number of successfully placed markers to the total number of empty grids in the initial environment.

G.4 SNAKE

In an 8 × 8 grid environment, the agent operates as the snake's head and seeks to consume (passthrough) as much food (markers) as possible without colliding with its own body. The snake's body grows by 1 grid cell for each marker consumption, and a new marker appears at a different location.

Throughout the episode, only one marker exists in the environment. The reward is defined as 1 20 , which is the reciprocal of the targeted number of marker consumption, for each marker consumption.

(d)

．．．．．．

(b)

(e)

(a)

(c)

(f)

回

回

向

向

H Details of KAREL-LONG problem set

We introduce the novel KAREL-LONG problem set as a benchmark to evaluate the capability of HIPO. Each task in our KAREL-LONG benchmark is crafted to exhibit long-horizon characteristics derived from Karel states. Additionally, we ensure that these tasks maintain a constant per-action cost, set at 0 . 0001 . Figure 16, Figure 17, Figure 18, Figure 19, and Figure 20 depict the tasks within the KAREL-LONG problem set, exhibiting randomly generated initial states and several internal states sampled from legitimate trajectories.

H.1 SEESAW

In a 16 × 16 grid environment, the agent's objective is to traverse back and forth between two 4 × 4 chambers (the left chamber and the right chamber), continuously collecting markers. To facilitate movement between the left and right chambers, the agent must traverse through a central 2 × 6 corridor. Initially, one marker is positioned in the left chamber, awaiting collection by the agent. Upon picking a marker in a chamber, another is randomly generated in the opposite chamber, continuing the agent's task of collecting markers. Consequently, the agent must navigate between the two chambers to collect markers continuously. The return is defined as the ratio of the number of picked markers to the total number of markers generated by the environment, termed as "emerging markers."

H.2 UP-N-DOWN

In an 8 × 8 grid, the agent's objective is to ascend and dscend the stairs repeatedly to collect markers (loads). Once a marker below/above the stairs is picked up, the next marker will appear above/below the stairs, and so on. The agent would receive an additional constant penalty (i.e., -0 . 005 ) for being out of contact with the stair. The return is defined as the ratio of the number of picked markers to the total number of markers generated by the environment, termed as "emerging loads."

H.3 FARMER

In an 8 × 8 grid environment, the agent's objective is twofold: initially, to fill the entire layout with markers, and subsequently, to collect them. In the initial state, all girds, except for the one in the upper-right corner, are empty, signaling the agent to proceed to populating the layout with markers, akin to farmer sowing seeds. After most grids are marked, the agent deploys the harvesting phase, analogous to a farmer collecting crops. Following this, the agent is prompted to refill the environment again, and the cycle repeats. We've imposed a maximum iteration limit to represent the anticipated number of filling-and-collecting rounds. The return is calculated as the ratio of the number of picked-and-placed markers to the total theoretical capacity of picked-and-placed markers, termed as "max markers."

H.4 INF-DOORKEY

In an 8 × 8 grid environment, partitioned into 4 chambers, the agent is tasked to pick up markers in certain chambers, place markers in others, and continuously traverse between chambers until a predetermined upper limit number of marker-picking and marker-placing is reached. As mentioned earlier, the environment is partitioned into 4 chambers, restricting the agent's placement (or pick-up) actions to one chamber at a time. Upon completing a placement (or pick-up) action in a chamber, the passage to the next chamber opens (an emptied grid cell), allowing the agent to advance and perform another action. The return is calculated as the ratio of the number of picked-and-placed markers to the total number of markers that the agent can theoretically pick and place, referred to as "max keys."

H.5 INF-HARVESTER

In a 16 × 16 grid environment, the agent's goal is to continuously pick up markers until none remain and no further markers emerge. Initially, the environment is fully stocked with markers. Each time the agent picks up a marker, there is a probability, termed the "emerging probability," that a new marker will appear in an empty grid cell, granting the agent to collect markers continuously and indefinitely. The return is calculated as the ratio of the number of picked markers to the expected total number of markers the environment can generate given a specific emerging probability.

I Designing domain-specific languages

Our program policies are designed to describe high-level task-solving procedures or decision-making logics of an agent. Therefore, our principle of designing domain-specific languages (DSLs) considers a general setting where an agent can perceive and interact with the environment to solve specific tasks. DSLs integrate control flows, perceptions, and actions. While control flows are domain-independent, perceptions and actions can be designed based on the domain of interest, requiring specific expertise and domain knowledge.

Such DSLs are proposed and utilized in various domains, including ViZDoom [34], 2D MineCraft [3, 71], and gym-minigrid [14]. Recent works [43, 79] also explored describing agents' behaviors using programs with function-taking arguments.

J Limitations

While the Hierarchical Programmatic Option (HIPO) Framework provides notable advantages including interpretability, long-horizon task solving, inductive generalizability, and sample efficiency, it also has limitations that merit consideration in comparison to other methods. Techniques utilizing first-order logic programming [19, 33, 50, 83], LLM-based interpretation [49], and decision tree policies [7, 35] are also specifically designed to create interpretable, explainable, and generalizable agents for solving MDPs.

In environments with a large state or action spaces, neuro-symbolic methods adopting symbolic policies [19, 33, 50, 83] are generally less computationally intensive and more feasible for training compared to programmatic approaches like this work and [10, 46, 74]. However, symbolic policies [19, 33, 50, 83] are inherently less intepretable, whereas programmatic policies, which could convey the logical flow of reasoning, are more intuitive to general users.

While these symbolic policies are often complex for the average user, INSIGHT [49] uses LLMs to explain symbolic policies (i.e. polynomials with trainable parameters), thereby enhancing interpretability. This is something we could consider for future improvements, like using LLMs to explain programmatic options for better interpretability. Nonetheless, LLM prompting requires detailed environment and task descriptions, where the generated explanation is dependent on the LLM's understanding of the environment and the quality of the descriptions. In contrast, HIPO directly derives an interpretable programmatic policy through interaction with the environment, avoiding the aforementioned issue.

Other proposals for enhancing interpretability involve decision trees. Both [7, 35] focus on learning decision trees within the imitation learning framework, which includes a DNN oracle for expert demonstration. In [35], learned decision trees are further converted into if-else Python programs. However, the interpretability and performance of these methods depend heavily on the tree learning algorithm. Overly complex or brute-force programs become less interpretable and meaningful. Moreover, these methods generally require specific state information (e.g. object coordinates) for predetermined training, making it challenging to identify the necessary symbolic features for optimal performance or to develop a plausible training scheme. In contrast, LEAPS and its successors search only within the embedding space and perceive the state space partially and consistently, operating with fewer assumptions and achieving good performance.

Besides, in our experiments detailed in Section 5, HIPO outperforms other baselines, including DRL, where deep neural networks were used as expert policies in [7, 35] to demonstrate trajectories for learning decision trees. This suggests that decision-tree methods [7, 35] are unable to surpass our HIPO framework due to the limitations of the underperforming DNN expert. In [7], the authors propose methods for verifying decision-tree policies. The structure of decision trees contributes to efficient verification, allowing for quantitative assessment of the policy's correctness, stability, and robustness. This process aims to evaluate the effectiveness of the policy, including its performance in various scenarios and the presence of other desirable properties. Developing a method to quantitatively verify a programmatic policy remains a significant challenge.

Finally, while HIPO retrieves a set of effective and compatible programs through a search method, exploring approaches that leverage prior knowledge, e . g ., large language models [47] or offline datasets in imitation learning [12, 28, 30, 38, 41, 60] or skill-based reinforcement learning [40, 54, 57], to efficiently retrieve programs is a promising research direction.

K Computational resources

For our experiments, we utilized the following workstation: 20-core Intel(R) Xeon(R) W-2255 CPU @3.70GHz, with 2X NVIDIA GeForce RTX 4070 Ti GPU

The rough computation time for the main experiments of our framework and baselines are as follows:

SEESAW

(a) DRL: 24 hours

(b) LEAPS: 48 hours

(c) HPRL: 48 hours

(d) HIPO: 48 hours

UP-N-DOWN:

(a) DRL: 24 hours

(b) LEAPS: 48 hours

(c) HPRL: 48 hours

(d) HIPO: 48 hours

FARMER:

(a) DRL: 24 hours

(b) LEAPS: 48 hours

(c) HPRL: 48 hours

(d) HIPO: 72 hours

INF-DOORKEY:

(a) DRL: 24 hours

(b) LEAPS: 48 hours

(c) HPRL: 48 hours

(d) HIPO: 72 hours

INF-HARVESTER:

(a) DRL: 24 hours

(b) LEAPS: 48 hours

(c) HPRL: 48 hours

(d) HIPO: 48 hours

L Impact statements

Our work introduces Hierarchical Programmatic Option framework (HIPO), a novel framework that combines programmatic RL and HRL, enabling autonomous agents to represent complex behaviors and address long-term tasks. HIPO demonstrates significant potential in automating laborious or risky tasks, enhancing the efficiency and safety of autonomous systems. While offering promising societal benefits, we acknowledge the importance of addressing ethical considerations, including potential biases inherited from training data or adversarial attacks. Ongoing research in the field of responsible AI and fairness in machine learning aims to mitigate these biases, and we acknowledge the importance of continuous efforts to ensure that AI systems, including HIPO, adhere to ethical standards and contribute positively to society.

(d)

．．．．．．

(b)

(e)

(a)

(c)

(f)

(g)

口

向

(a)

(b)

(d)

(c)

(e)

回

宣

(d)

．．．．．．

(b)

(e)

(a)

(c)

(f)

(g)

向

向

向

向

回

(d)

．．．．．．

(b)

(e)

(a)

(c)

(f)

回

向

(a) DOORKEY

(b) ONESTROKE

(c) SEEDER

(d) SNAKE

Karel Programs

SEESAW

Option 1

DEF

run

m(

move

move

pickMarker

turnLeft

WHILE

c(

frontIsClear

c)

w(

move

pickMarker

w)

WHILE

c(

frontIsClear

c)

w(

move

pickMarker

w)

m)

Option 2

DEF

run

m(

WHILE

c(

markersPresent

c)

w(

pickMarker

turnRight

w)

WHILE

c(

markersPresent

c)

w(

pickMarker

w)

WHILE

c(

markersPresent

c)

w(

pickMarker

w)

m)

Option 3

DEF

run

m(

IF

c(

frontIsClear

c)

i(

turnLeft

move

move

pickMarker

turnLeft

IF

c(

frontIsClear

c

)

i(

move

pickMarker

move

pickMarker

i)

move

i)

IF

c(

frontIsClear

c)

i(

pickMarker

move

pickMarker

i)

move

pickMarker

m)

UP-N-DOWN

Option 1

DEF

run

m(

IF

c(

frontIsClear

c)

i(

move

move

i)

IF

c(

not

c(

leftIsClear

c)

c)

i(

move

move

i)

m)

Option 2

DEF

run

m(

turnLeft

move

turnRight

move

m)

Option 3

DEF

run

m(

turnLeft

move

turnRight

move

turnLeft

move

turnRight

move

turnLeft

move

turnRight

move

turnLeft

move

turnRight

move

turnLeft

move

turnRight

move

turnLeft

move

turnRight

move

turnLeft

move

turnRight

move

turnLeft

move

turnRight

move

turnLeft

move

turnRight

move

turnLeft

move

turnRight

move

m)

Karel Programs

FARMER

Option 1

DEF

run

m(

pickMarker

REPEAT

r(

pickMarker

move

turnRight

move

move

turnLeft

move

move

turnRight

move

turnLeft

move

move

pickMarker

move

turnLeft

move

move

pickMarker

move

turnLeft

move

move

pickMarker

move

r)

m)

Option 2

DEF

run

m(

turnRight

move

putMarker

turnRight

move

pickMarker

putMarker

move

pickMarker

putMarker

move

pickMarker

putMarker

move

pickMarker

putMarker

move

pickMarker

putMarker

move

m)

Option 3

DEF

run

m(

turnRight

move

turnRight

putMarker

pickMarker

move

pickMarker

putMarker

move

pickMarker

putMarker

move

pickMarker

putMarker

move

pickMarker

putMarker

move

pickMarker

putMarker

move

pickMarker

m)

Option 4

DEF

run

m(

REPEAT

R=17

r(

IF

c(

not

c(

rightIsClear

c)

c)

i(

putMarker

move

i)

r)

m)

Option 5

DEF

run

m(

putMarker

pickMarker

turnRight

move

pickMarker

turnRight

move

pickMarker

putMarker

move

pickMarker

putMarker

move

pickMarker

putMarker

move

pickMarker

putMarker

move

pickMarker

putMarker

move

m)

INF-DOORKEY

Option 1

DEF

run

m(

IFELSE

c(

noMarkersPresent

c)

i(

move

putMarker

i)

ELSE

e(

pickMarker

putMarker

e)

m)

Option 2

DEF

run

m(

IF

c(

noMarkersPresent

c

)

i(

move

move

move

IF

c(

noMarkersPresent

c)

i(

move

REPEAT

R=1

r(

move

turnLeft

r)

i)

i)

IF

c(

noMarkersPresent

c

)

i(

move

REPEAT

R=1

r(

move

REPEAT

R=1

r(

move

r)

r)

i)

m)

Option 3

DEF

run

m(

putMarker

pickMarker

IF

c(

not

c(

markersPresent

c)

c

)

i(

move

i)

WHILE

c(

noMarkersPresent

c)

w(

IFELSE

c(

markersPresent

c)

i(

turnLeft

i)

ELSE

e(

move

turnRight

e)

w)

m)

Option 4

DEF

run

m(

putMarker

turnLeft

pickMarker

move

turnLeft

IFELSE

c(

markersPresent

c)

i(

move

turnLeft

i)

ELSE

e(

move

e)

m)

Option 5

DEF

run

m(

turnRight

IF

c(

leftIsClear

c)

i(

turnLeft

i)

putMarker

IF

c(

leftIsClear

c)

i(

turnLeft

i)

pickMarker

move

IF

c(

leftIsClear

c)

i(

turnLeft

i)

putMarker

move

IF

c(

leftIsClear

c)

i(

turnLeft

i)

pickMarker

move

m)

INF-HARVESTER

Option 1

DEF

run

m(

REPEAT

R=7

r(

turnLeft

move

REPEAT

R=8

r(

turnLeft

WHILE

c(

frontIsClear

c)

w(

pickMarker

move

w)

r)

turnLeft

REPEAT

R=8

r(

pickMarker

r)

r)

m)

Option 2

DEF

run

m(

WHILE

c(

frontIsClear

c)

w(

IF

c(

frontIsClear

c

)

i(

IF

c(

frontIsClear

c)

i(

pickMarker

move

i)

IF

c(

frontIsClear

c)

i(

pickMarker

i)

i)

w)

IF

c(

frontIsClear

c)

i(

pickMarker

i)

pickMarker

m)

Option 3

DEF

run

m(

REPEAT

R=5

r(

turnLeft

move

pickMarker

move

turnLeft

pickMarker

move

pickMarker

move

pickMarker

move

pickMarker

move

pickMarker

move

pickMarker

move

pickMarker

move

pickMarker

move

pickMarker

move

pickMarker

move

pickMarker

move

pickMarker

move

pickMarker

move

pickMarker

move

pickMarker

r)

m)

NeurIPS paper checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the claims made.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The details are shown in Section J.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The details are shown in Section E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release the code as soon as possible.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The details are shown in Section E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports the means and variances of the experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The compute resources are shown in Section K.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The statements are shown in Section L.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The original publications that produced the assets are cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This work builds off of deep reinforcement learning generating programmatic policies, and adapts it to solve long-horizon, repetitive tasks. Concretely, this work proposes HIPO, short for hierarchical programmatic option framework, which retrieves history programs by its neural embeddings, and uses these programs as reusable options to solve reoccurring tasks.

STRENGTHS:
1) Effective approach: This work tackles an intuitively challenging problem and proposes a practically effective solution. Particularly on long-horizon tasks, as shown by the results in Karel-long in Table 2, demonstrates the effectiveness of the proposed method.
2) Thorough ablation studies: the paper conducts various ablation studies, and show the optimal performance of the proposed approach among other variants.

WEAKNESSES:
1) Lack of retrieval evaluation: The paper proposes special techniques to improve the effectiveness, diversity, and compatibility of retrieved programs, however, is there any empirical evidence to show that the designed modules do improve the effectiveness, diversity, and compatibility of programs?

QUESTIONS:
1) While generating reusable subroutines is intuitive, there are no examples on how these low-level policies look like throughout the paper, making it a bit harder to concretely visualize how the policies look like. Adding a few examples, even in appendix, would make the process more interpretable.
2) This work chooses a particular DSL (as introduced in Section 3), does the choice of DSL affect the performance of the proposed process? Would a less well-designed DSL cause the method to be ineffective?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUtilizing human-readable programs as policies has been recently proposed to enhance interpretability in reinforcement learning. This work introduces the Hierarchical Programmatic Option Framework (HiPO) that first embeds the programs into a smooth and continuously parametrized space, obtains a diverse and compatible skill set by applying the cross entropy method on the embedding space with respect to novel diversity and compatibility metrics, and finally learning a high-level policy upon the skill set. Experimental analysis of HiPO on the Karel problem sets manifests its effectiveness and zero-shot generalizability.\n\nSTRENGTHS:\nThe authors proposed novel reward functions that can enhance the diversity and compatibility of the program skill set obtained by CEM. The experimental results show that HiPO performs better on average compared to existing baselines, and every part of HiPO plays a significant role.\n\nWEAKNESSES:\n1. The execution time will differ from program to program, but HiPO does not consider this factor while incorporating a discount factor $\\gamma$ of 0.99. The adoption of SMDP should be considered to deal with the variance in execution time appropriately. Also, HiPO does not discount the cumulative reward of low-level actions, further deepening the inconsistency between the theoretical objective and the actual loss function. The authors should justify such design choices.\n\n2. In Section 5.3, the difference between the two settings, CEM+diversity top k and CEM+diversity x |M|, is unclear. Further explanation would be helpful for the readers.\n\n3. The authors did not conduct experiments on the CEM + compatibility setting. In the Karel-Long environments, most of the improvement seems to come from the compatibility heuristic.\n\nQUESTIONS:\n1. Figure 6 shows how the adoption of the diversity factor spreads the skill set over the embedding space. However, the diversity of the embedding vectors does not necessarily guarantee the diversity in the actual behaviour. How different are the programs' actual behaviors in the final skill set?\n\n2. From Section A.2, it seems like at every step, the procedure selects one program through CEM. Wouldn't this approach cause the programs selected in the earlier stages to be sub-optimal?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present HIPO, a method that use a Program embedding space to create options. Some are retrieve if diverse and efficient to create a set of option, later used by a learned high level policy. To evaluate their framework, as the KAREL benchmark does not include long and repetitive tasks, they introduce KAREL-LONG and evaluate their method on both benchmarks. Their result show that HIPO is a viable hierarchical RL method.\n\nSTRENGTHS:\nThe paper is sound and clear. The structure is clear, the contributions easily identifiable, the figures, table are easily understandable. \nI am not a deep expert of option or program synthesis, but the paper is so clear that it brought me sufficient insight to situate it and understand its method. The KAREL-LONG test-bench is well motivated, as one can see that the original framework is not enough to differentiate its existing SOTA methods. \nTheir evaluation shows that HIPO helps with long and repetitive tasks, while being the strongest competitor on the existing KAREL domain. \nI'm already giving an accept, but might further raise based on the other reviews and the rebuttal.\n\nWEAKNESSES:\nThe only main weakness I was able to identify is the:\n\n**Lack of limitation section**. A dedicated *Limitation* section could clearly help situate the advantages and drawback of HIPO, and compare it to other methods. Authors could step a little bit outside programmatic and look into e.g. interpretable logic-based RL [1, 2, 9, 10], interpretable RL methods that use LLM [3], which can be useful for programs synthesis, tree-based policies (convertible to tree programs)[8, 5, 6, 7], ... etc. I think the \"background\" part of current related work section could be included in the *Introduction*, and that a dedicated *Limitation and related work* section could include such a discussion. It would situate the method in the broader literature, and bring attention to the nice HIPO method to these communities as well. I give some pointers references, but some more could be searched. \n\nFurther smaller concerns are: \n\n**Interpretability allows to detect invisible suboptimal behaviors.** It has been shown that non-interpretable (e.g. deep) RL methods often learn misaligned policies [4, 5, 6], without this problem being spotted, on tasks as simple as Pong [5]. This is a much stronger argument to directly place in the introduction than the lack of transparency and trust. I would start off using such an argument, highlighting the need of e.g. readable, interpretable programs. \n\n**Authors could slightly modify your method name (if they wish).** Your methods' name might collide with *Hierarchical Proximal Policy Optimization (HIPPO)*, an existing known option learning framework. \n\n-------------------------------\n[1] Jiang, Zhengyao, and Shan Luo. \"Neural logic reinforcement learning.\" International conference on machine learning. PMLR, 2019.\n\n[2] Xu, Duo, and Faramarz Fekri. \"Interpretable model-based hierarchical reinforcement learning using inductive logic programming.\" arXiv preprint arXiv:2106.11417 (2021).\n\n[3] Luo, Lirui, et al. \"INSIGHT: End-to-End Neuro-Symbolic Visual Reinforcement Learning with Language Explanations.\" arXiv preprint arXiv:2403.12451 (2024).\n\n[4] Di Langosco, Lauro Langosco, et al. \"Goal misgeneralization in deep reinforcement learning.\" International Conference on Machine Learning. PMLR, 2022.\n\n[5] Delfosse, Quentin, et al. \"Interpretable concept bottlenecks to align reinforcement learning agents.\" arXiv preprint arXiv:2401.05821 (2024).\n\n[6] Kohler, Hector, et al. \"Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning.\" arXiv preprint arXiv:2405.14956 (2024).\n\n[7] Delfosse, Quentin, et al. \"HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning.\" arXiv preprint arXiv:2406.03997 (2024).\n\n[8] Bastani, Osbert, Yewen Pu, and Armando Solar-Lezama. \"Verifiable reinforcement learning via policy extraction.\" Advances in neural information processing systems 31 (2018).\n\n[9] Delfosse, Quentin, et al. \"Interpretable and explainable logical policies via neurally guided symbolic abstraction.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[10] Ma, Z., Zhuang, Y., Weng, P., Zhuo, H. H., Li, D., Liu, W., & Hao, J. (2021). Learning symbolic rules for interpretable deep reinforcement learning. arXiv preprint arXiv:2103.08228.\n\nQUESTIONS:\n* Are the KAREL-LONG environment containing training and testing environments ? As programs are compared, it might be insightful to test the generalizability of the different methods (by varying size, of the environment).\n* I am not sure if the option is returning a terminal token or if the high level controller is reselecting an option at each timestep. This could be made clearer. \n* What's the percentage of retrieved program (based on your effectiveness and diversity criterion) ?\n\nLIMITATIONS:\nThis constitutes my main concern, I have not been able to clearly situate a (broader discussion) on the limitations. Again, I would create such a section, bringing most of the related work in it, as an opportunity to criticize (both positively and negatively) HIPO, and other approaches.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents HIPO, a method for learning programmatic options for solving problems with long-horizon and repetitive tasks. First, HIPO searches in the space defined by a domain-specific language for a set of diverse programs. This set of programs is generated while accounting for the diversity and composability of the programs. Finally, these programs are used as options for a neural policy. HIPO was evaluated on instances of Karel the Robot, where it was shown to be more sample efficient than Deep RL and other programmatic methods. The paper also includes a carefully designed ablation study on HIPO's components.\n\nSTRENGTHS:\nThe idea of using programmatic representations to learn options is interesting and valuable. The authors correctly state that using programmatic policies as options as opposed to policies means that one sacrifices, to some extent, the interpretability of the policies in favor of performance.\n\nAnother strength of HIPO is its natural combination of programmatic and neural representations -- this is not mentioned in the paper and perhaps it should! The repetitive behavior required to solve the tasks is given by the programmatic options, which can achieve this through the strong inductive bias the Karel language offers. The neural policy is then responsible for orchestrating these different components, which would clearly be difficult through program synthesis.\n\nI am particularly impressed by the results on the DoorKey domain. This is such a difficult domain for program synthesis and DRL. HIPO managed to achieve a really high score in that domain. I do have some clarifying questions about this result (please see them in the Questions section of this review).\n\nI also appreciated the extra effort the authors put into translating the neural model that orchestrates the options into automata for interpretability. Since the automata can also be seen as programs, it would be interesting to see their performance or even their translation into the Karel language.\n\nWEAKNESSES:\nThe paper has a few weaknesses too. It would be helpful if the authors managed to fix the following for the camera ready, if the paper is accepted.\n\n1. HIPO represents a method for learning programmatic options, but it includes no option-learning baseline.\n2. Searching directly in the programmatic space was recently shown to outperform the latent search used in HIPO. It would be valuable to include this type of search as a baseline too.\n3. It would be interesting to measure sample efficiency in terms of samples instead of programs. Intuitively, some programs could run much longer and be more costly in real settings. This would also allow for a comparison with DRL in terms of sample complexity. In general, I find learning curves more informative than the tables presented in the paper, which show only asymptotic performance.\n4. It would be valuable to have more runs of the experiments >30 and show confidence intervals instead of standard deviation, as CI is easier to draw conclusions from without statistical tests.\n\nOverall, in my opinion, the strengths of the paper outweigh its weaknesses. This is because the paper shows a different way of using programmatic representations that the community should further explore.\n\nQUESTIONS:\nPlease feel free to comment on the points listed under weaknesses. In addition to them, I will add the following questions and comments.\n\n1. I am puzzled by the outstanding performance of HIPO in DoorKey. The way that the options are selected is that they have to be diverse and they have to maximize the agent's return. Since the options are selected as a pre-processing step, before orchestrating them, the programs must be collecting the key and receiving a positive reward, all of them. Once they are orchestrated, they happen to contain the behavior needed to find the door and collect the marker after the key is found. Is it because the composability step is already finding a solution that stitches together options that are able to solve the problem? If that is the case, would it be valuable to add a baseline that performs diversity+composability and returns the best order used in the composability step?\n\n2. Why does HIPO use an option that terminates the episode? This doesn't make sense to me. The episode finishes whenever it finishes. That is, it is not up to the agent to decide when the episode finishes. This is a property of the MDP.\n\n3. The caption of Figure 3 says that the Harvester goes from 36 to 400 markers as we move from Harvester to Inf-Harvester. Due to the structure of the problem, isn't it possible to find a short program that is able to collect an arbitrary number of markers?\n\n4. I would like to challenge the desiderata of the options (lines 136-139). First, an option doesn't have to be effective in the sense that it needs to \"obtain some task reward.\" It just needs to be part of the solution. For example, in the DoorKey problem, if the agent only received a positive reward once the problem was solved, an option that collects the key would still be helpful. Second, I suspect that the property of compatibility is only needed because the neural agent doesn't have access to the primitive actions while orchestrating the options. If the agent had access to them, it would be able to combine any \"helpful option\" by using primitive actions in between them. However, I understand that even if the agent had access to primitive actions, it could still find value in compatibility. In this case, compatibility would be playing the effectiveness property, and it would be looking at ways of how different programs can be combined to maximize the agent's return.\n\n5. How can the diversity score be defined for searching directly in the programmatic space? Could one use the losses used to train the latent space to define diversity? Or is the diversity score only possible because of the properties of the latent space?\n\nLIMITATIONS:\nThe paper has a limitations section in the appendix that addresses some important limitations, such as the trade-off between interpretability and performance. I would also add the points raised in the Weaknesses section of this review and the use of a single domain in the experiments.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}