Programmatic Reinforcement Learning without Oracles

1 INTRODUCTION . A growing body of research has explored programs in a domain-specific programming language as a new RL policy representation that intentionally encourages policy interpretability . Yet , learning a policy as a high-level program in structured representations is challenging . This is because algorithms must jointly identify a reasonable program architecture to allow for sufficient expressiveness while optimizing the parameters of a program ’ s modules . For example , depending on the shape of a maze , walking a robot to different goals on the maze by a program may require various if-then-else conditions to travel along different paths , the number of which might not be known to the agent before training . Existing programmatic policy synthesis algorithms either learn from a pretrained program embedding space that must support smooth interpolation ( Trivedi et al. , 2021 ) , or must be guided by the supervision of a pretrained oracle ( e.g . a neural network policy trained by RL ) via a teacher-student learning paradigm ( Bastani et al. , 2018 ; Silver et al. , 2020 ; Inala et al. , 2020 ; Verma et al. , 2018 ; 2019 ) . The task of imitating an oracle is , intuitively , significantly simpler than the full RL problem . However , since the policy spaces of neural networks and that of programmatic policies are significantly different , a significant performance gap exists between an imitating program and its RL oracle ( Verma et al. , 2018 ) . The first contribution of our paper is a framework to instead synthesize interpretable and differentiable programmatic policies solely from reward signals by policy gradient methods , without needing any oracles and pretraining . A conceivable way of synthesizing program architectures would be to enumerate all possible architectures induced by the grammar of a domain-specific policy language , run standard RL for each to find the optimal values of its unknown parameters , and return the best program . However , doing so is computationally expensive as each RL trial may explore millions of environment steps . Inspired by recent advances in differentiable neural architecture search e.g . DARTs ( Liu et al. , 2019b ) , we relax the discrete program architecture search space to be continuous . Specifically , we encode program architecture synthesis as learning the probability distribution over all possible architecture derivations ( up to a certain bound on program abstract syntax tree depth ) induced by a policy language grammar . This enables our RL algorithm to jointly optimize program architectures and the parameters of program modules via policy-gradient methods . Our second contribution is improving programmatic policies to support compositionality — the integration of primitive functions trained to grasp basic , task-agnostic skills ( e.g . running forward or jumping ) into a new complex function as a composite model to solve novel RL problems ( e.g . jumping over multiple hurdles to reach a target ) . As opposed to policy ensemble models based on neural networks ( Qureshi et al. , 2019 ) , our programmatically composite models interpret how primitive functions are composed under different environment conditions based on an RL agent ’ s perceptions , and naturally generalize to novel scenarios . We further apply programmatic policies to address challenging hierarchical RL problems . Our solution leverages the specifications of primitive functions to create an optimal high-level control plan via a satisfiability constraint solver and implements the high-level plans by learned composition of primitive functions . Finally , we benchmark our method against the state-of-the-art RL methods . Our results demonstrate that the programmatic RL framework is able to solve extremely hard RL-problems using highly interpretable policies with improved task performance . 2 PROBLEM MOTIVATION AND FORMULATION . We study how to express RL policies as differentiable programs , which use symbolic language constructs to compose a set of parameterized primitive modules . To control an agent , a programmatic policy takes an environment state as input and computes an action as return for the agent to execute . We view a programmatic policy as a pair ( E , θ ) , where E is a discrete program architecture and θ is a vector of real-valued parameters of the program . A program architecture E is structured based on the context-free grammar ( Hopcroft et al. , 2007 ) of a policy DSL . In this paper , we consider the context-free grammar depicted in the standard Backus-Naur form ( Winskel , 1993 ) in Fig . 1 . A vertical bar “ | ” indicates choice . Such a grammar consists of a set of production rules X : := σ1 σ2 · · · σj where X is a nonterminal and σ1 , · · · , σj are either terminals or nonter- minals . For example , we may expand the nonteriminal E1 in a partial program if B1 then C1 else E1 to if B1 then C1 else ( if B2 then C2 else E2 ) . The nonterminals E and B stand for program expressions that evaluate to action values in Rm and Booleans , respectively , where m is the action dimension size . We represent a state input to a programmatic policy as s = { x1 : ν1 , x2 : ν2 , . . . , xn } where n is the state dimension size and νi = s [ xi ] is the value of xi in s. As usual , the unbounded variables in X = [ x1 , x2 , . . . , xn ] are assumed to be input variables ( state variables in our context ) . A terminal in this grammar is a symbol that can appear in a program ’ s code , e.g . the if symbol and xi . The semantics of a program in E is mostly standard and given by a function JEK ( s ) , defined for each DSL construct . For example , JxiK ( s ) = s [ xi ] reads the value of a variable xi in a state input s. A policy may use an if-then-else branching construct . To avoid discontinuities for differentiability , we interpret its semantics in terms of a smooth approximation where σ is the sigmoid function : Jif B then C else EK ( s ) = σ ( JBK ( s ) ) · JCK ( s ) + ( 1− σ ( JBK ( s ) ) · JEK ( s ) ( 1 ) Thus , any policy programmed in this grammar becomes a differentiable program . C is a controller used by a programmatic policy . During execution , the policy can invoke a set of controllers under different environment conditions , according to the activation of B conditions in the program . We consider three DSLs depending on how C is structured for affine , ensemble , and PID policies . Affine Policies . The DSL for affine policies allows C to be expanded as an affine transformation : CAffine : := θc + θ · X | θc where θ ∈ Rm·|X | , θc ∈ Rm are control parameters . Particularly , CAffine can be as simple as some ( learned ) constants θc . An example affine policy is given in Appendix K.1 . if θ1c + θ T 1 · X > 0 then ( 95 % · πUP ( s ) + 5 % · πLEFT ( s ) ) ←− Branch 1 else if θ2c + θ T 2 · X > 0 then ( 95 % · πLEFT ( s ) + 5 % · πRIGHT ( s ) ) ←− Branch 2 else ( 13 % · πDOWN ( s ) + 87 % · πRIGHT ( s ) ) ←− Branch 3 X = [ x , y , Gx , Gy , arctan yx , ‖x , y‖2 ] θ1 = [ − 2.052 , 0.049 , 0.440 , 0.181 , 0.241 , 1.443 ] , θ1c = −0.202 θ2 = [ 1.333 , 2.204 , −2.2171 , 2.132 , 1.878 , 0.331 ] , θ2c = −0.416 Figure 3 : An Ant Cross Maze program Pcross with three branches . A program input X includes current Ant position x , y along with the target location Gx , Gy ( sampled from one of the three goals in Fig . 2 ) . arctan yx and ‖x , y‖2 are functions of x and y . Each branch composes primitive functions : πUP , πDOWN , πLEFT , and πRIGHT . Composition weights are shown in percentage . Ensemble Policies . The most important feature of our programmatic model is compositionality — composing and reusing task-agnostic primitives in new programs to solve novel problems . The DSL for ensemble policies includes pre-acquired primitives π1 , · · · , πN as callable library functions : Cπ : := θ1 · π1 ( s ) + θ2 · π2 ( s ) + · · ·+ θN · πN ( s ) Cπ explicitly compose primitive functions ( e.g . running forward or jumping ) hierarchically into a complex program ( e.g . jumping over multiple hurdles to reach a target ) where θ1 , · · · , θN ∈ R1 parameterize a primitive combination . The input space of a primitive function can be different than that of the composite program ( formally defined below ) . Its semantics is captured as follows : JCπK ( s ) = N∑ i=0 qi · πi ( s ) where qi = exp ( θi/T ) ∑N j=0 exp ( θj/T ) Here the composition weights { qi } Ni=0 for primitive ensemble are computed using gumbel-softmax , where T is the temperature term ( Jang et al. , 2017 ) . PID Policies . Suppose we know a priori that PID control is suitable for stabilising of an RL system . We can express this knowledge using the DSL for PID functions that allows C to be expanded as discretized , multivariable PID controllers ( Zheng et al. , 2002 ) . We leave the details in Appendix K.2 . Program Interpretability . Our RL algorithm searches over a DSL to synthesize a programmatic policy . Thanks to the structured and symbolic representations , it learns highly interpretable policies . For example , consider an Ant Cross Maze environment depicted in Fig . 2 . The cross-maze contains three possible goal positions and one would be randomly selected at each time . In this environment , the task for a quadruped MuJoCo Ant is to reach the selected location by navigating through the maze staring from an initial position on the bottom and without collision or crash . We consider the DSL for this task using ensemble policies Cπ . It includes four basic primitive functions for moving the Ant up πup , down πdown , left πleft , and right πright ( pretrained as neural network policies using standard RL algorithms with details left in App . F.2 ) . Fig . 3 depicts a synthesized program Pcross with three branches for solving the Ant Cross Maze environment . As specified in Equation 1 , our semantics of a branching construct is approximated by the sigmoid function σ . The value of the predicate in a Boolean condition determines the activation of the controller guarded by the Boolean condition . At each state , branch activation determines the strength of each of the controllers in the program . For example , the activation of branch 1 is σ ( θ1c + θ T 1 · X ) , and the activation of branch 2 is ( 1− σ ( θ1c + θT1 · X ) ) · σ ( θ2c + θT2 · X ) . Fig . 4a depicts the activation of branch 1 as a function of ( x , y ) when the goal to reach is sampled at Gx = 12 , Gy = 0 . The degree of activation ( yellow ) is close to 1 on all states under ( 12 , 0 ) indicating that the ensemble policy at branch 1 is used to drive the Ant up to the goal . Indeed , according to the distribution of each primitive function at branch 1 , the effect of πUP dominates . Fig . 4b , Fig . 4c , and Fig . 4d depict the activation of all three branches when the goal is at Gx = 6 , Gy = −6 . The program can be interpreted as branch 1 ( where πUP dominates ) and branch 3 ( where πRIGHT dominates ) are activated in the yellow areas of Fig . 4b and Fig . 4d respectively . This allows the Ant to make a curved up and right move to the goal ( branch 2 is not activated during execution for this goal ) . Problem Formulation . We frame programmatic RL as a Markov Decision Process ( MDP ) defined by a tuple { S , A , T , R } where S and A represent the environment state space and action space , T : S × A × S → [ 0 , 1 ] captures the set of transition probabilities , and R : S × A → R denotes the reward function . We assume S ⊇ R|X∪V| where X is the set of input variables of a composite program ( defined by a DSL ) and V is the set of input variables of primitive functions . For a noncomposite affine policy , V = ∅ . At time t ≥ 0 , an RL agent receives an environment state st ∈ S and performs an action at ∈ A selected by its policy π ( at|st ) : S → A . Based on st and at , the agent transits to receive the next state according to the transition model T ( st+1|st , at ) , and receives the reward R ( st , at ) . We aim to learn a programmatic policy π in the DSL in Fig . 1 by jointly synthesizing the program ’ s architecture E and optimizing the program ’ s parameters θ to maximize the cumulative discounted reward Es0 , a0 , s1···∼π [ ∑∞ 0 γ t ·R ( st , at ) ] where γ ∈ ( 0 , 1 ] .

Summary: This paper presents a novel method for synthesizing programmatic policies. The code idea of the method is to define a relaxed and differentiable version of the domain-specific language (DSL) used to encode the programmatic policies.   The program space the DSL induces can be described as a program of the DSL itself. Since the program defines a differentiable space, one can use policy gradient methods to search in the space of programs by assigning higher probabilities to production rules that maximize the expected rewards.   The search for programmatic policies happens in two steps. The first step applies a policy gradient method to optimize the probabilities defining the program space. Then, one can extract the most likely program structure from the search space by greedily choosing the production rules with higher probabilities. The resulting program is further optimized with reinforcement learning.   Empirical results on continuous control problems show the advantages of the method.