Programmatic Reinforcement Learning without Oracles

1 INTRODUCTION . A growing body of research has explored programs in a domain-specific programming language as a new RL policy representation that intentionally encourages policy interpretability . Yet , learning a policy as a high-level program in structured representations is challenging . This is because algorithms must jointly identify a reasonable program architecture to allow for sufficient expressiveness while optimizing the parameters of a program ’ s modules . For example , depending on the shape of a maze , walking a robot to different goals on the maze by a program may require various if-then-else conditions to travel along different paths , the number of which might not be known to the agent before training . Existing programmatic policy synthesis algorithms either learn from a pretrained program embedding space that must support smooth interpolation ( Trivedi et al. , 2021 ) , or must be guided by the supervision of a pretrained oracle ( e.g . a neural network policy trained by RL ) via a teacher-student learning paradigm ( Bastani et al. , 2018 ; Silver et al. , 2020 ; Inala et al. , 2020 ; Verma et al. , 2018 ; 2019 ) . The task of imitating an oracle is , intuitively , significantly simpler than the full RL problem . However , since the policy spaces of neural networks and that of programmatic policies are significantly different , a significant performance gap exists between an imitating program and its RL oracle ( Verma et al. , 2018 ) . The first contribution of our paper is a framework to instead synthesize interpretable and differentiable programmatic policies solely from reward signals by policy gradient methods , without needing any oracles and pretraining . A conceivable way of synthesizing program architectures would be to enumerate all possible architectures induced by the grammar of a domain-specific policy language , run standard RL for each to find the optimal values of its unknown parameters , and return the best program . However , doing so is computationally expensive as each RL trial may explore millions of environment steps . Inspired by recent advances in differentiable neural architecture search e.g . DARTs ( Liu et al. , 2019b ) , we relax the discrete program architecture search space to be continuous . Specifically , we encode program architecture synthesis as learning the probability distribution over all possible architecture derivations ( up to a certain bound on program abstract syntax tree depth ) induced by a policy language grammar . This enables our RL algorithm to jointly optimize program architectures and the parameters of program modules via policy-gradient methods . Our second contribution is improving programmatic policies to support compositionality — the integration of primitive functions trained to grasp basic , task-agnostic skills ( e.g . running forward or jumping ) into a new complex function as a composite model to solve novel RL problems ( e.g . jumping over multiple hurdles to reach a target ) . As opposed to policy ensemble models based on neural networks ( Qureshi et al. , 2019 ) , our programmatically composite models interpret how primitive functions are composed under different environment conditions based on an RL agent ’ s perceptions , and naturally generalize to novel scenarios . We further apply programmatic policies to address challenging hierarchical RL problems . Our solution leverages the specifications of primitive functions to create an optimal high-level control plan via a satisfiability constraint solver and implements the high-level plans by learned composition of primitive functions . Finally , we benchmark our method against the state-of-the-art RL methods . Our results demonstrate that the programmatic RL framework is able to solve extremely hard RL-problems using highly interpretable policies with improved task performance . 2 PROBLEM MOTIVATION AND FORMULATION . We study how to express RL policies as differentiable programs , which use symbolic language constructs to compose a set of parameterized primitive modules . To control an agent , a programmatic policy takes an environment state as input and computes an action as return for the agent to execute . We view a programmatic policy as a pair ( E , θ ) , where E is a discrete program architecture and θ is a vector of real-valued parameters of the program . A program architecture E is structured based on the context-free grammar ( Hopcroft et al. , 2007 ) of a policy DSL . In this paper , we consider the context-free grammar depicted in the standard Backus-Naur form ( Winskel , 1993 ) in Fig . 1 . A vertical bar “ | ” indicates choice . Such a grammar consists of a set of production rules X : := σ1 σ2 · · · σj where X is a nonterminal and σ1 , · · · , σj are either terminals or nonter- minals . For example , we may expand the nonteriminal E1 in a partial program if B1 then C1 else E1 to if B1 then C1 else ( if B2 then C2 else E2 ) . The nonterminals E and B stand for program expressions that evaluate to action values in Rm and Booleans , respectively , where m is the action dimension size . We represent a state input to a programmatic policy as s = { x1 : ν1 , x2 : ν2 , . . . , xn } where n is the state dimension size and νi = s [ xi ] is the value of xi in s. As usual , the unbounded variables in X = [ x1 , x2 , . . . , xn ] are assumed to be input variables ( state variables in our context ) . A terminal in this grammar is a symbol that can appear in a program ’ s code , e.g . the if symbol and xi . The semantics of a program in E is mostly standard and given by a function JEK ( s ) , defined for each DSL construct . For example , JxiK ( s ) = s [ xi ] reads the value of a variable xi in a state input s. A policy may use an if-then-else branching construct . To avoid discontinuities for differentiability , we interpret its semantics in terms of a smooth approximation where σ is the sigmoid function : Jif B then C else EK ( s ) = σ ( JBK ( s ) ) · JCK ( s ) + ( 1− σ ( JBK ( s ) ) · JEK ( s ) ( 1 ) Thus , any policy programmed in this grammar becomes a differentiable program . C is a controller used by a programmatic policy . During execution , the policy can invoke a set of controllers under different environment conditions , according to the activation of B conditions in the program . We consider three DSLs depending on how C is structured for affine , ensemble , and PID policies . Affine Policies . The DSL for affine policies allows C to be expanded as an affine transformation : CAffine : := θc + θ · X | θc where θ ∈ Rm·|X | , θc ∈ Rm are control parameters . Particularly , CAffine can be as simple as some ( learned ) constants θc . An example affine policy is given in Appendix K.1 . if θ1c + θ T 1 · X > 0 then ( 95 % · πUP ( s ) + 5 % · πLEFT ( s ) ) ←− Branch 1 else if θ2c + θ T 2 · X > 0 then ( 95 % · πLEFT ( s ) + 5 % · πRIGHT ( s ) ) ←− Branch 2 else ( 13 % · πDOWN ( s ) + 87 % · πRIGHT ( s ) ) ←− Branch 3 X = [ x , y , Gx , Gy , arctan yx , ‖x , y‖2 ] θ1 = [ − 2.052 , 0.049 , 0.440 , 0.181 , 0.241 , 1.443 ] , θ1c = −0.202 θ2 = [ 1.333 , 2.204 , −2.2171 , 2.132 , 1.878 , 0.331 ] , θ2c = −0.416 Figure 3 : An Ant Cross Maze program Pcross with three branches . A program input X includes current Ant position x , y along with the target location Gx , Gy ( sampled from one of the three goals in Fig . 2 ) . arctan yx and ‖x , y‖2 are functions of x and y . Each branch composes primitive functions : πUP , πDOWN , πLEFT , and πRIGHT . Composition weights are shown in percentage . Ensemble Policies . The most important feature of our programmatic model is compositionality — composing and reusing task-agnostic primitives in new programs to solve novel problems . The DSL for ensemble policies includes pre-acquired primitives π1 , · · · , πN as callable library functions : Cπ : := θ1 · π1 ( s ) + θ2 · π2 ( s ) + · · ·+ θN · πN ( s ) Cπ explicitly compose primitive functions ( e.g . running forward or jumping ) hierarchically into a complex program ( e.g . jumping over multiple hurdles to reach a target ) where θ1 , · · · , θN ∈ R1 parameterize a primitive combination . The input space of a primitive function can be different than that of the composite program ( formally defined below ) . Its semantics is captured as follows : JCπK ( s ) = N∑ i=0 qi · πi ( s ) where qi = exp ( θi/T ) ∑N j=0 exp ( θj/T ) Here the composition weights { qi } Ni=0 for primitive ensemble are computed using gumbel-softmax , where T is the temperature term ( Jang et al. , 2017 ) . PID Policies . Suppose we know a priori that PID control is suitable for stabilising of an RL system . We can express this knowledge using the DSL for PID functions that allows C to be expanded as discretized , multivariable PID controllers ( Zheng et al. , 2002 ) . We leave the details in Appendix K.2 . Program Interpretability . Our RL algorithm searches over a DSL to synthesize a programmatic policy . Thanks to the structured and symbolic representations , it learns highly interpretable policies . For example , consider an Ant Cross Maze environment depicted in Fig . 2 . The cross-maze contains three possible goal positions and one would be randomly selected at each time . In this environment , the task for a quadruped MuJoCo Ant is to reach the selected location by navigating through the maze staring from an initial position on the bottom and without collision or crash . We consider the DSL for this task using ensemble policies Cπ . It includes four basic primitive functions for moving the Ant up πup , down πdown , left πleft , and right πright ( pretrained as neural network policies using standard RL algorithms with details left in App . F.2 ) . Fig . 3 depicts a synthesized program Pcross with three branches for solving the Ant Cross Maze environment . As specified in Equation 1 , our semantics of a branching construct is approximated by the sigmoid function σ . The value of the predicate in a Boolean condition determines the activation of the controller guarded by the Boolean condition . At each state , branch activation determines the strength of each of the controllers in the program . For example , the activation of branch 1 is σ ( θ1c + θ T 1 · X ) , and the activation of branch 2 is ( 1− σ ( θ1c + θT1 · X ) ) · σ ( θ2c + θT2 · X ) . Fig . 4a depicts the activation of branch 1 as a function of ( x , y ) when the goal to reach is sampled at Gx = 12 , Gy = 0 . The degree of activation ( yellow ) is close to 1 on all states under ( 12 , 0 ) indicating that the ensemble policy at branch 1 is used to drive the Ant up to the goal . Indeed , according to the distribution of each primitive function at branch 1 , the effect of πUP dominates . Fig . 4b , Fig . 4c , and Fig . 4d depict the activation of all three branches when the goal is at Gx = 6 , Gy = −6 . The program can be interpreted as branch 1 ( where πUP dominates ) and branch 3 ( where πRIGHT dominates ) are activated in the yellow areas of Fig . 4b and Fig . 4d respectively . This allows the Ant to make a curved up and right move to the goal ( branch 2 is not activated during execution for this goal ) . Problem Formulation . We frame programmatic RL as a Markov Decision Process ( MDP ) defined by a tuple { S , A , T , R } where S and A represent the environment state space and action space , T : S × A × S → [ 0 , 1 ] captures the set of transition probabilities , and R : S × A → R denotes the reward function . We assume S ⊇ R|X∪V| where X is the set of input variables of a composite program ( defined by a DSL ) and V is the set of input variables of primitive functions . For a noncomposite affine policy , V = ∅ . At time t ≥ 0 , an RL agent receives an environment state st ∈ S and performs an action at ∈ A selected by its policy π ( at|st ) : S → A . Based on st and at , the agent transits to receive the next state according to the transition model T ( st+1|st , at ) , and receives the reward R ( st , at ) . We aim to learn a programmatic policy π in the DSL in Fig . 1 by jointly synthesizing the program ’ s architecture E and optimizing the program ’ s parameters θ to maximize the cumulative discounted reward Es0 , a0 , s1···∼π [ ∑∞ 0 γ t ·R ( st , at ) ] where γ ∈ ( 0 , 1 ] .

Summary: This paper addresses the problem of the low efficiency of program search guided by a pre-trained oracle or on discrete and non-differentiable architecture space. To this end, the paper proposes a framework that performs program architecture search on top of a differentiable relaxation of the architecture space. This allows the program architectures and parameters to be learned via policy-gradient methods without RL oracle. The proposed method also exploits compositionality by allowing an ensemble of primitive functions that perform task-agnostic skills. The experimental results on navigation and manipulation domains show that the proposed method can reliably obtain task-solving programs and outperforms or performs competitively compared to RL baselines including SAC, PPO, TRPO. I believe this work studies an interesting and promising research direction and proposes a convincing framework to tackle this problem with solid technical contributions. Yet, I am mainly concerned with some missing baselines, relevant works, and ablations.