Efficient Symbolic Policy Learning with Differentiable Symbolic Expression
Authors: Jiaming Guo
Rui Zhang
Shaohui Peng
Qi Yi
Xing Hu
Ruizhi Chen
Zidong Du
Xishan Zhang
Ling Li
Qi Guo
Yunji Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: reinforcement learning
context variables
symbolic policy

Abstract: Deep reinforcement learning (DRL) has led to a wide range of advances in sequential decision-making tasks. However, the complexity of neural network policies makes it difficult to understand and deploy with limited computational resources. Currently, employing compact symbolic expressions as symbolic policies is a promising strategy to obtain simple and interpretable policies. Previous symbolic policy methods usually involve complex training processes and pre-trained neural network policies, which are inefficient and limit the application of symbolic policies. In this paper, we propose an efficient gradient-based learning method named Efficient Symbolic Policy Learning (ESPL) that learns the symbolic policy from scratch in an end-to-end way. We introduce a symbolic network as the search space and employ a path selector to find the compact symbolic policy. By doing so we represent the policy with a differentiable symbolic expression and train it in an off-policy manner which further improves the efficiency. In addition, in contrast with previous symbolic policies which only work in single-task RL because of complexity, we expand ESPL on meta-RL to generate symbolic policies for unseen tasks. Experimentally, we show that our approach generates symbolic policies with higher performance and greatly improves data efficiency for single-task RL. In meta-RL, we demonstrate that compared with neural network policies the proposed symbolic policy achieves higher performance and efficiency and shows the potential to be interpretable.

Efficient Symbolic Policy Learning with Differentiable Symbolic Expression
Jiaming Guo Rui Zhang' Shaohui Peng* Qi Yi Xing Hu
Ruizhi Chen Zidong D Xishan Zhang!4 Ling L
Quo Yunji Chen
SKI of Processors, Institute of Computing Technology, CAS, Beijing, China
2 Intelligent Software Research Center, Institute of Software, CAS, Beijing, China
University of Science and Technology of China, USTC, Hefei, China Cambricon Technologies
5 Shanghai Innovation Center for Processor Technologies, SHIC, Shanghai, China University of Chinese Academy of Sciences, UCAS, Beijing, China
(guojiaming, zhangrui)@ictt. ac. en, lengshaohui@iscas ac en, yiqi@mail ustc.. eed.. huringlict. ac cn, ruizhi@iscas ac. en {duzidong, whanggisaannrnnaaii..... ac..cn liling@iscas. cn.cn, {guoqi, (yjo)i,cy))ee cyj) cn
Abstract
Deep reinforcement learning (DRL) has led a wide range of advances sequen- tial decision-making tasks. However, the complexity of neural network policies makes it difficult to understand and deploy with limited computational resources. Currently, employing compact symbolic expressions as symbolic policies is a promising strategy to obtain simple and interpretable policies. Previous symbolic policy methods usually involve complex training processes and pre- pre-trained neural network policies, which are inefficient and limit the application of symbolic poli- cies. In this paper, propose efficient gradient-based based learning method named Efficient Symbolic Policy Learning (ESPL.) that learns the symbolic policy from scratch in end-to-end way. We introduce symbolic network the search space and employ a path selector to find the compact symbolic policy. By doing so we represent the policy with differentiable symbolic expression and train it in off-policy manner which further improves the efficiency. In addition, in contrast with previous symbolic policies which only work in single-task RL because of complexity, we expand ESPL on meta-RL to generate symbolic policies for unseen tasks. Experimentally, we show that our approach generates symbolic policies with higher performance and greatly improves data efficiency for single-task RL. In meta-RL, we demonstrate that compared with neural network policies the pro- posed symbolic policy achieves higher performance and efficiency and shows the potential be interpretable.
Introduction
With the development of deep neural networks as general-purpose function approximators, deep reinforcement learning (DRL) has achieved impressive results in solving sequential decision-makinn tasks [ [2]. In DRL, the policies are commonly implemented as deep neural networks which involve tremendous parameters and thousands of nested non-linear operators. Despite the excellent representation ability, the neural network (NN) is very complex, making it difficult to understand, predict the behavior and deploy with limited computational resources.
"Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

With the high academic and industrial interest in interpretable and simple RL policy, some works propose to learn the symbolic policy which a symbolic expression composing variables, constants, and various mathematical operators. The symbolic policy has succinct form and low complexity which is considered to be more interpretable and easily deployable real-world settings. 3 and [4] approximate symbolic policy with genetic programming but are limited to simple tasks with provided or learned world model and suffered from performance decrease compared with NN policies. DSP [S). the state-of-the art symbolic policy learning method, removes some limitations by employing a recurrent neural (et) as an agent to generate the symbolic policy and trains the RNN with reinforcement learning. However, this method has low data efficiency and requires hundreds of times of environment interactions compared with traditional reinforcement learning algorithms. For enviromments with multidinensional action spaces, they need pre- pre-trained NN policy as the anchor model, which brings additional complexity and may limit the final performance. In addition, the complexity of these algorithms makes it difficult apply them complex reinforcement learning tasks, e.g. meta-reinforcement learning.
In this paper, we propose an efficient tradient-based learning method called ESPL (Efficient Symbolic Policy Learning) for learning the symbolic policy from scratch in an end-to-end differentiable way. To express the policy in symbolic form, the proposed ESPL consists of a symbolic network and path selector. The symbolic network can be considered as a full set of candidate symbolic policies. In the symbolic network, the activation functions are composed of various symbolic operators and the parameters can be regarded as the constants in the symbolic expression. The path selector chooses the proper compact symbolic form from the symbolic network by adaptively masking out irrelevant connections. We design all these modules to be differentiable and represent the policy with differentiable symbolic expression. Then can efficiently train the symbolic policy in an off-policy manner and the symbolic policy is directly updated via gradient descent without an additional agent. Experimentally, on several benchmark control tasks, our algorithm is able to produce pelf-oefrnning symbolic policy while requiring thousands of times fewer environmental interactions than DSP.
Meta-reinforcement learning (meta-RL) is one of the most important techniques for RL applications, which improves the generalization ability on unseen tasks by learning the shared internal structure across several tasks. We raise the question: is it possible to exploit the benefit of the symbolic policy and meta-RL to generate symbolic policies for unseen tasks? While previous symbolic policy methods too complex to be combined with meta-RL., we combine the proposed ESPL with context-based meta-RL and develop the contextual symbolic policy (CSP). Context- based me 6 7 g [9]
is the most promising meta-RL method which forces the policy to be conditional on context variables that are formed by aggregating experiences. In the CSP framework, the path selector decides the symbolic form based the context variables. We also involve a parameter generator to generate the constants of the symbolic policy based on the context variables. We build the training process on top of the context-based meta-RL method PEARL (7). The proposed CSP can generate symbolic policies for unseen tasks given few trajectories. We find that compared with neural network policies, contextual policies produced by CSP achieve higher generalization performance, efficiency, and show the potential to be interpretable.
The contributions of this paper are three-fold. First, introduce a novel gradient-based symbolic policy learning algorithm named ESPL that learns the symbolic policy efficiently from scratch. Next, with ESPL we develop the contextual symbolic policy for meta-RL, which can produce symbolic policies for unseen tasks. Finally, we summarize our empirical results which demonstrate the gain of ESPL both in single-task RL and meta-RL. Importantly, we find empirically that contextual symbolic policy improves the generalization performance in PEARL.
2 Related Works
2.1 Symbolic Policy
The emergence of symbolic policy is partly credited to the development of symbolic regression which is applicable in wide fields, e.g. discovering physics lows [00] and automated CPU design [. Symbolic regression aims to find symbolic expressions to best fit the dataset from an unknown fixed function. A series of methods 12 13 [4] employ genetic programming (GP) to evolve the symbolic expressions. With the development of neural network and gradient descent, some methods
[i5] 16 [7] involve deep learning for symbolic regression. Some works employ symbolic regression
2

---

Paan
Example network structures for the symbolic network. Left: the plain structure. Middle: a symbolic work with dense connections. Right: a symbolic network with dense connections and arranged operators.
Figure 1:
methods to obtain symbolic policies for efficiency and interpretability. 3 and 4 aim to approximate symbolic policy with genetic programming but require given dynamics equations a learned world model. DSP [5), following the symbolic regression method DSR [16]. employs a recurrent neural network generate the symbolic policy. They use the average returns of the symbolic policies as the reward signal and train the neural network with risk-seeking policy gradients. However, for environments with multidimensional action spaces, they need pre-trained neural network policy as the anchor model. Besides, in this framework, a single reward for reinforcement learning involves many environmental interactions, which is inefficient and makes hard to combine the symbolic policy with meta-RL. Recently, some works [8. 19 attempt to distill an interpretable policy from a pre-trained neural network policy but have a problem of objective mismatch [5]. Different from the above-mentin meneeneet methods, we propose an efficient gradient-based based framework to obtain the symbolic policy without any pre-trained model.
2.2 Meta-Reinforceemnt Learning
Meta-RL extends the notion of meta-learning 20 21 22] to the context of reinforcement learning. Some works 23 24 25 aim to meta-learn the update rule for reinforcement learning. We here consider another research line of works that meta-train a policy that can be adapted efficiently to a new task. Several works [26 27 28 learn initialization and adapt the parameters with policy gradient methods. However, these methods are inefficient because of the on-policy learning process and the gradient-based updating during adaptation. Recently, context-based meta-RL. 6
7 29 achieve higher
efficiency and performance. For example, PEARL 77) proposes an off-policy meta-RL. method that infers probabilistic context variables with experiences from environments. Hyper 29) proposes hypernetwork where the primary network determines the weights of conditional network and achieves higher performance. Most of the subsequent context-based meta-RL methods [30] 31 32
attempt to achieve higher performance by improving the context encoder or the exploration strategy. In this paper, we combine the symbolic policy with meta-RL to form the CSP and consequently improve the efficiency, interpretability and performance of meta-RL. As far as we know, we are the first learn the symbolic policy for meta-RL.
Our method is also related to some neural architecture search methods and programmatic RL methods. We provide extended literature review in Appendix E
3 Gradient-based Symbolic Policy Learning
This section introduces the structure of the proposed ESPL, an end-to-end differentiable system. The proposed ESPI.cossist of two main components: 1) the Symbolic Network, which expresses the policy symbolic form, and 2) the Path Selector, which selects paths from the symbolic network to form compact symbolic expressions.
3

---

3.1 Densely Connected Symbolic Network
To construct symbolic policy in an end-to-end differentiable form, we propose the densely connected symbolic network SN as the search space for symbolic policies. Inspired by previous differentiable symbolic regression methods [77 [3]. we employ a neural network with specifically designed units, which is named symbolic network. We now introduce the basic symbolic network named plain structure which is illustrated in T The symbolic network is feed-forward network with T. layers. Different from traditional neural networks, the activation functions of the symbolic network are replaced by symbolic operators, e.g. trigonometric and exponential functions. For the in layer of the symbolic network, we denote the input as æ1-1 and the parameters weights wy and biases by. These parameters serve as the constants a symbolic expression. We assume that the 4 layer contains unary functions {gi.... I } and binary functions {gi,-..,,h Firstly, the input of the 4n layer will be linearly transformed by a fully-comnected layer:
y1 11) wy + bi-
The fully- layer realizes the addition and subtraction in symbolic expressions and produces m- + 2n outputs. Then the outputs will go through the symbolic operators and be concatenated to form the layer output:
(3 (ot) Ihh (vi"), 9. (1 .1 2
(y.
1/ 2n]]
(2)
Then the ln layer of the symbolic network can be formulated as SN x G((F((n--)). Following the last layer, a fully-connected layer will produce single output. For multiple action dimensions, we construct symbolic network for each dimension of action.
The symbolic operators are selected from a library. e.g. {sin, eop, log, x,+} for continuous control tasks. For the plain structure, we include an identical operator which retains the output of the previous layer to the next layer in the library. To find the symbolic policy via gradient descent, it is critical ensure the numerical stability of the system. However, this is not natural in a symbolic network. For example, the division operator and the logarithmic operator will create pole when the input goes to zero and the exponential function may produce a large output. Thus, we regularize the operators and employ penalty term to keep the input from the "forbidden" area. For example, the logarithmic operator y (a) returns log(r) for x > bounding and log (boundhog)) otherwise and the penalty term is defined as fiog mar (bourduogg xr 0). The division operator e a/b returns a/b for b > bounddie and 0 otherwise. The penalty term is defined as Lait maz regularized operators can be found in the Appendix. To ensure the numerical stability, we involve a penalty loss function Cpenaity which is the sum of the penalty terms of all N regularized operators in symbolic networks:
Symbolic operator.
- b, 0). The details of all
im
Epenalty
Ca (2x).
Dense connectivity. We introduce dense connections 34 in the symbolic network, where inputs of each layer are connected to all subsequent layers. Consequently, the leh layer of the symbolic network will receive the environment state and the output of all preceding layers 21, -21-1 an ([s,, (I........... I)) The dense connections improve the information flow between layers and benefit the training procedure. Besides, with these dense skip connections across layers, the combination symbolic operators becomes more flexible, making the symbolic network more likely to contain good symbolic policies. In addition, we can flexibly arrange the position of operators. For example, if we only arrange the sin operator in the last layer but the oracle expression contains terms like sin( so). the input of the sin operator can still be from the original state because of the dense connections. We give an example arranged operators in Figure | which we use for all tasks in the experiments. In this symbolic network, we heuristically involve more multiplication and division operators at shallow layers to provide more choice of input processed by simple operators for complex operations such sines and cosines.
3.2
The Path Selector
The symbolic network serves as full set of the search space of symbolic expressions. To select the proper paths from the symbolic network to produce compact symbolic policy, we reduce the

---

number of paths involved in the final symbolic policy then proper paths remain and redundant paths are removed. This can be naturally realized by minimizing the Lo norm of the symbolic network employ Tn norm instead of Tog norm. However, I will penalize the magnitude of the weights and result performance degradation. Inspired by the babbability-basedd sparsification method 35 36
weights. As the Ta norm is not differentiable, some methods 17 33
37. we propose
probabilistic path selector which selects paths from the network by multiplying a binary mask on the weights of the symbolic network w. The binary mask m, is sampled from the Bernoulli distribution: my ~ (ern(p:), where p; E [0, 1] serves as the probability. Then the final weights of the symbolic X m, where X is the element -isee multiply operation. Consequently, to
network are Wm get a compact symbolic expression, we only need to minimize the expectation of the Lo norm of the binary mask Em Bern(m p) m mmll Σ Pt. without penalizing the magnitude of the weights. During the process of collecting data or testing, we can directly sample the binary mask from the Bernoulli distribution. Then we can obtain the symbolic policy Tsym by removing paths with zero weight and simplifying the symbolic expression.
However, the sampling process does not have a well-defined gradient. Thus, for the training process we build up our sampling function with the gumbel-softmax trick [38]. As the mask m is binary categorical variables, we replace the softmax with sigmoid and named the sampling function as gumbel sigmoid. The gumbel sigmoid function can be formulated as:
p mgg sigmoid( log( 1-p + 91 90
T
where 91 and go are iid samples drawn from Gumbel 1). T is the temperature annealing parameter. Note that mgs is still not a binary mask. To obtain binary mask but maintain the gradient, we employ the Straight-Through (ST) trick: m = 120.s(Ms= + mgg= mgge. where 120.5(z) E {0, 1 is the indicator function and the overline means stopping the gradient. During training, we do not remove paths with zero weight and directly use symbolic network S.Wwwm) as the policy.
We also involve loss function Ceelect to regularize the sum of probabilities p which is the expectation of the Lo norm of the binary mask m. To limit the minimum complexity of symbolic policies, we involve the minimum Lo norm defined as Imin. Then the loss function can be defined as:
Lesetect max Σ Pt Imini 0).
3.3 Implementation
In practice, we build our off- learning framework on top of the soft actor-critic algorithm (SAC) [39]. We employ the neural network Q(s,a) parameterized by 0g the critic function). To construct a stochastic policy, we also employ a small neural network F(s) parameterized by OF to output the standard deviation. Note that Q(s, a) and F(s) are only used during training. We optimize the weights w, the biases b of the symbolic network, the probabilities in the path selector, and Or with the combination of actor loss from SAC, Lepenatty and L.ctect- We update Oo with the critic loss from SAC. During training, we decrease the temperature parameter of gumbel sigmoid linearly and decrease the Lmin from the count of the original parameters w to a target value with parabolic function. We summarize the training procedure and give the pseudo-code in Appendix C
4
Contextual Symbolic Policy for Meta-RL
4.1
Background
In the field of meta-reinforeemmnt learning (meta-RL), we consider a distribution of tasks p(k) with each task i ~ p(t)) modeled as a Markov Decision (rocess(MDP)) In common meta-RL settings, tasks share similar structures but differ in the transition and/or reward function. Thus, we can describe a task K with the 6-tuple (S, A. Pre. Po. ?). In this setting, S C R" is set of n-dimensional states, A C R. is a set of m-dimensional actions, P.- : S x A S + [o, 1] is the state transition probability distribution, Po S - [0, 1] is the distribution over initial states, Tx S x A + R is the reward function, and 7 E (0, 1) is the per timestep discount factor. Following the setting of prior works 7 8). we assume there are M meta-trai training tasks /kmmm=1.....k .M sampled from the training
5

---

Table 1: Symbolic policies produced by ESPL.
Environment
ESPL
CartPole
17.17
1.2sg
MountainCar
61
8.06min (9.3x22 0.18) 1.26
Pendulum
d1 (4.27s:
0.62) (1.92 0.42
InvDoublePend
12.30ss
4.48sin 2 4.518;
2366 7.97ss 1.238g 0.08) 0.34
Inv PendSwingup
01
1.76
0.33s,
0.29) 0.65
01 = (0.14 2.57
0.68log
(0.xx)
1.44
LunarLander
42
5.72sy + 4.44stm 0.03)
0.4
-6.5 13 78.in(4. -0.00000000000 0.06)
4.71
Q1 -033211 - 1.468 0.83810
5013 2.57 + 0.38) 0.92
Hopper
= -00-0..0 0.52> 3.6384 4.58sg 0.68e=y(( 7.31
2. + 0.58 + 62 3.000
1.33 4aai
0.83 + 1.16s 0.47 1e
-1.
88 56 1.32)
123113 50+ 0.088
0
01 = 1.45 2.94cos( 0. 73 + (0.06 1.00
33 0.28 66 + 0.41) 1) + 1.32)
a2 7.53ezp( 00.48) 13s, 0.52sin
(1.5
0.24)
11.1
Bipedal Walker
1.070-0041
0.55
4.60m (0.2 3.01gg
0.03
0.90
04 0.28 + 3.20
Table Performance comparison symbolic policies and neural policies for seven different DRL
algorithms.
Environment
DDPG
TRPO
A2C
PPO
ACKTR
SAC
TD3
Regression
DSP
ESPL
Carppoee
1000
1000
1000
1000
1000
1000
1000
211.82
1000
1000
Mountaincat
95.36
93.6
93.97
93.76
93.79
94.68
93.87
95.16
99.11
94.02
Pendulum
-155.6
-145.49
-157.59
-160.14
-201.57
-154.82
-155.06
-1206.9
-155.4
-151.72
DnDooulleendd
9
9188.43
9359.81
9356.59
9359.06
9359.92
9359.25
637.2
9149.9
9359,9
Inv PendeSwingup
891.48
892.9
254.71
890.1
890.11
891.32
892.25
-19.21
891.9
890.36
LunarLander
266.05
265.26
238.51
269.65
271.53
276.92
272.13
56.08
261.36
283.56
Hopper
1678.84
2593.56
2104.98
2586.56
2583.88
2613,16
2743.9
47.35
2122.4
2442.48
BipedalWalker
209.42
312.14
291.79
287.43
309.57
308.31
314.24
-110,77
311.78
309.43
Worst Rank
10
9
9
10
6
Average
6.125
5.375
3.5
tasks distribution Ptrain (x). For meta-testing, the tasks sampled from the test tasks distribution (k). The two distributions are usually the same in most settings but can be different in out- (O-distribution(OODD) settings. We denote context er {($1, 01, r1,,)), (ST, ( S FT } as the collected experiences. For context-based meta-RL, the agent encodes the context into a the
latent context variable with context encoder (=]cc) and the policy is conditioned current state and the context variable z. During adaptation, the agent first collects experiences for a few episodes and then updates the context variables. Then the contextual policy is able to adapt to new tasks according to the context variables. The meta-RL objective can be formulated max Expp((s) (erpr [R(s, n, ([][c))]]], where R(s, n, ((2ler) denotes the expected episode return.
4.2 Incorporating the Context Variables
To quickly adapt to new tasks, we need to incorporate the context variables 2 q(2(cx) to the symbolic network and produce different symbolic policies for different tasks K sampled from the task distribution p(x). To condition the parameters of the symbolic expression on the context variable, we propose a parameter generator: w. b = +==) which is neural network to produce the parameters of symbolic networks for all action dimensions based on the context variables. We also involve a neural network to generate the probabilities of the path selector: p = (22). Then the contextual symbolic network can generate different symbolic expression forms according to the context variables.

---

Hopper-
Walkerr2d
Cheetah aod
return
return
return
Myper
CSP
20M
6200
3798
12W
123
1004
3354
1.001
1.04
2004
2304
1000
X284
C2R8
0.04
1.00
1304
1714 2201
env steps
env steps
env Steps
Lunariander
InvooublePend params
Cartpole-fl fl-ond
return
-30
3050
3.500
12:00
1.5M0
0.000
1.802
CNV stops
Cnv steps
CNv steps
Figure 2: Comparison for different kinds of contextual policies on meta-RL tasks. We show the mean and standard deviation of returns on test tasks averaged over five runs.
4.3 Training Schedule
We train the CSP in an off-policy manner. For meta-training epoch t, the agent first collects experiences of different training tasks into the corresponding buffer Bx, for several iterations. At the beginning of each data collection iteration, we sample context cr from buffer Bx, and sample context variables z ~ g==(eT) as PEARL [7] does. The difference is that we also sample the symbolic policy with (==) and (=) and use the sampled policy for the following steps of the iteration. For training, we sample RL batch and context from the buffer and optimize the context encoder q(zler) to recover the state-action value function. For each training step, we sample new symbolic policy. We employ the soft actor-critic to optimize the state-action value function. For the parameter generator and the path selector, employ Leetect and Epenatty in addition to the SAC loss. During training, we decrease the temperature parameter T and 4min just like single-task RL. More details and the pseudo-code can be found in Appendix
5 Experiment
5.1 Experimental Settings
Environment. For single-task RL, we evaluated our method on benchmark control tasks which are presented DSP: (1) CartPole; (2) MountainCar; (3) Pendulum; (4) InvertedDoubleeenduunn
(5) EnvertePeendulumSSingge we evaluate the CSP on several continuous control environments which are modified from the environments of OpenAl Gym [40 to be meta-RL tasks similar to 7 9 30 These environments require the agent to adapt across dynamics (random system parameters for Hopper-params, Walker2d- params, aaarllader--pramms Cartpole-fl-oo or reward functions (target velocity for vel-ood..
(6) LunarLander (7) Hopper; (8) BipedalWalker. For meta-RI. RL,
HeDDDoleeeeddppraateee different force magnitude and pole length for
Methods. In the single-task RL experiments, for the neural network policies, we compare our method with seven state-of-the-ar DRL algorithms: DDPG, TRPO, A2C, PPO, ACKTR, SAC, and TD3 The results are obtained with the tuned pretrained policies from an open- open-source repository Zoo [47] For symbolic policies, we include the Regression method and DSP. The Regression policies are produced with two steps: 1) generate a dataset of observation action trajectories from the best pre-trained policy from Zoo; 2) perform deep symbolic regression [16] on the dataset and select the expression with the lowest emror for each action. DSP first trains recurrent neural network with reinforcement learning to produce symbolic policies and then optimizes the constant with several methods such as Bayesian Optimization (48).
41
39
46
7

---

Table 3: The number of environment episodes used for learning symbolic policies.
Environment
Regression
DSP
ESPI
CartPole
1000
500
MountainCar
1000
2M
500
Pendulum
1000
2M
500
Inv DuvulleePend
1000
2M
500
Inv PendSSiiiguup
1000
2M
500
LunarLander
1000
0.4M
1000
Hopper
1000
0.4M
3000
SipedalWalkes
1000
0.4M
2000
Evaluation. Following DSP, we evaluate the proposed ESPL by averaging the episodic rewards across 1 1,000 episodes with different environment seeds. The evaluation for all the baselines is also in accordance with this protocol. For the DSP and regression method, we use the results from the DSP paper. DSP performs 3n independent training runs for environments with n-dimension action and select the best symbolic policy. For a fair comparison, we perform three independent runs and select the best policy for ESPL and DRL methods. For meta-RL, we run all environments based the off-ppolicy meta-learning framework proposed by PEARL (7] and use the same evaluation settings. We compare CSP with PEARL which concatenates the observation and context variables as the input of policy and Hyper [29] which generate the parameters of policy with a ResNet model based on the context variables. Note that the original Hyper also modifies the critic, but we build all the critics with the same network structure for consistency. More details of the experimental settings can be found in Appendix D
5.2 Comparisons for Single-task RL
Performance. In Table 2 we report the average episode rewards of different algorithms. Among symbolic policies, the proposed ESPL achieves higher comparable performance compared with DSP while the regression method performs poorly in most environments. To directly compare ESPL with other algorithms across different environments, rank and worst-case rank. We find that TD3 and SAC outperform other algorithms. For symbolic policy methods, the proposed ESPL achieves superior performance compared DSP. ESPL is also comparable with the better-performing algorithms of DRL.
rank the algorithms and caleulate the average
Data efficiency. In Table 3 report the number of environment episodes required for learning the symbolic policy in different algorithms. The proposed ESPL requires fewer environmental interactions to learn the symbolic policies in most environments. Compared with DSP, the proposed ESPL uses thousands of times fewer environment episodes Although the regression method requires similar number of environment episodes ESPL, it fails find meaningful policies in most environments according to Table Besides, the proposed ESPL is trained from scratch while the regression method and DSP need pretrained neural network policies. In conclusion, the proposed ESPL greatly improves the data efficiency of symbolic policy learning.
5.3 Comparisons for Meta-RL
Performance. For meta-RL tasks, report the learning curves of undiscounted returns the test tasks in Figure 2 We find that CSP achieves better or comparable performance in all the environments compared with previous methods. In Hopper- paramm, Walker2d- paramms and Inv Dnuuleeeeendd-aat -aramms, CSP outperforms PEARL and Hyper during the whole training process. In Lunarlander-paramt CSP achieves better final results. In Cartpole-fl-ood CSP adapts to the optimal more quickly. In the out-of-distribution task Cheetah-vel-ood. we find the performance of PEARL and Hyper decrease during training because of over-fitting. But our CSP is less affected. Thus, expressing the policy in the symbolic form helps improve the generalization performance.
Deploying efficiency. We also evaluate the deploying efficiency of contextual policies. We first calculate the flops of each kind policy per inference step. Then we consider an application scenario
For DSP, we only record the environment episodes required for reinforcement learning as the exact number of iterations for constant optimization is not provided in DSP.

---

Table 4: FLOPs (k)/Inference time (ms) of different contextual policies.
Environment
CSP
PEARL.
Hyper
paaaersdsparams
3.11/20.9
189.3/27.0
5.64/22.6
Hopper-parane params
0.51/4.13
186.9/26.6
4.1/17.2
Inv DDoublePenddpeeeee -araams
0.039/0.37
186.0/25.1
3,59/12.3
Caripole-fl-o Lunarlander-g Chectah-vel-oug
0.004/0.042
183.9/23.9
1.79/9.08
0.015/0.14
185.4/23.4
3.08/12.3
0.53/4.9
190.2/28.4
7.18/24.2
that the algorithm control five thousand simulated robots with the Intel(R) Xeon(R) Gold 5218R 2.10GHz CPU and record the elapsed time per inference step. We report the results in Table 4 Compared to PEARL, CSP reduces the FLOPs by 60-45000x and reduces the inference time by up to 600x. Compared to Hyper, CSP reduces the flops by 2-450x and reduces the inference time by up to 200x. Thus, compared with pure NN policies, the CSP has significant advantage in computational efficiency.
5.4 Analysis of symbolic policies
Interpretability. For single-RL. tasks, we report the symbolic policies found by ESPL for each environment in Table [ The policies in the symbolic form are simple and we can directly glean insight from the policies by inspection, or in other words, interpretability. For example, the goal of LunarL-ander to land a spacecraft on landing pad without crashing. The action a1 controls the main engine and controls the orientation engines. In our symbolic policy, aj is function of 81 (the height) and 84 (the vertical velocity). The main engine turns on to counteract downward motion when the height is low. Action 02 is combination of term about syhe horizontal velocity), term about sthe angle), and highly nonlinear term about s6 (the angular velocity), 81 (the horizontal position) and 57 (whether the leg has landed). Thus, the policy adjusts the orientation engines based on the incline and horizontal motion to move the spacecraft to the center of the landing pad. In meta-RL RL, the CSP also shows the potential be interpretable. We take the Cartpole- fl-ood environment example and illustrate the Cartpole system in Figure 3 The form of the symbolic policies produced by CSP action = cl * + C2 * + b, where cl and c2 are the positive coefficients and b is a small constant which can be ignored. Then the policy direction that the pole is deflected or will be deflected. To analyze the difference between policies for different tasks, we uniformly symbolic policy with CSP and record the coefficients. As Figure shows, el and c2 tend to increase when the force magnitude decrease and the length increase, which is in accord with our intuition. We also provide the human study results of the interpretability in Appendix F3
be interpreted as pushing the cart in the
the force magnitude and the length of the pole. Then we generate the
Complexity. we compare the length of the symbolic policies and define length EE N where n is the dimension of the action, is the index of the action dimension, Nf the number of operators, N is the number of constant terms, Ná. is the number of variable terms. We give comparison the length of the symbolic policies in Table
Table 5: Comparison of the length of the symbolic policies.
Average
CartPole
MountainCar
Pendulum
Lnv Doubleeendd
PendendSwingup
LunerLander
Hopper
BipedalWalker
ESPL 12.91 DSP
16.5
24.6
17
8.25
19
6.5
12
12,5
In the benchmark environments used in the literature, in some cnvironments ESPL produces longer symbolic policies than DSP, in others ESPL produces similar or shorter symbolic policies than DSP. In general, symbolic policies produced by ESPL are only slightly longer than the symbolic policies produced by DSP.
5.5 Ablation
Finally, we carry out experiments by ablating the features of the proposed ESPL. We change the structure of the symbolic network and replaced the path selector with the L, norm minimization. We

---

13
0.6
05
6.
04
Figure 3: The Cartpole system be Figure 4: The coefficients of symbolic policies for Cartpole environ- ments with different force magnitude and pole length.
controlled.
Table 6: Ablation results for single-task task RL. ESPLp and ESPLj means replacing the symbolic network with plain structure and densely connected structure respectively. ESPL,, means replacing the path selector with the I norm minimization.
Environment
ESPL
ESPLp
ESPLa
ESPL
CartPole
1000
1000
1000
1000
MountainCar
94.02
93.69
93.83
94.17
Pendulum
-15.772
-183.16
-144.08
-63.554
InvooublePend
9359.9
9357.6
9197.44
8771.18
InvPendS wingup
890.36
844.84
890.01
865.38
Lander
283.56
263.95
277.36
271.69
Hopper
2442.48
2003.24
2316.54
1546.35
Bipedal
BipedalWalker Walker
309.43
-11.63
298.50
6.81
report the average episode rewards for single-task RL in T As the experiment results show, without the path selector the dense connections, the performance degrades, especially in Hopper and BipkdalWalker. With the path selector or the dense connections, ESPLy is able to perform well for all the environments while we observe that the arranged symbolic operators can further improve the performance. We also provide the ablation study results for meta-RL R in Appendix F2
6 Conclusions
In this paper, we introduce ESPL, an efficient gradient-bad basedd symbolic policy learning method. The proposed ESPL is able to learn the symbolic policy from scratch in an end-to-end way. The experiment results on eight continuous control tasks demonstrate that the approach achieves comparable higher performance than both NN-based policies and previous symbolic policies while greatly improving the data efficiency compared with the previous symbolic policy method. We also combine our method with meta-RL to generate symbolic policies for unseen tasks. Empirically, compared with neural network policies, the proposed symbolic policy achieves higher performance and efficiency and shows the potential to be interpretable. We hope ESPL can inspire future works of symbolic policy for reinforcement learning or meta-reinforcemeet learning. Besides, as symbolic policy a white box and more dependable, the proposed ESPL may promote applications of reinforcement learning in industrial control and automatic chip design.
Limitations and future work. In this paper, we focus on continuous control tasks with low- dimensional state space. The proposed ESPL and CSP can not directly generate a symbolic policy for tasks with high-dimensional observation like images. A possible method is to employ neural network extract the environmental variables and generate symbolic policy based on these en- vironmental variables. We leave this in the future work. Symbolic policies generally have good interpretability. However, when task too complex, the symbolic policy also more complex, making the interpretability decrease. Solving this problem is also an important direction for future work. For application, we will further learn symbolic policy for automated CPU design based on this framework to optimize the performanncepooerraare (PPA) of the CPU.
10

---

7 Acknowledgement
This work is partially supported by the National Key R&D Program of China (under Grant 20212D01 10102). the NSF of China (under Grants 61925208, 62102399, 62222214, 62002338, U22A2028, U19B2019), CAS Project for Young Scientists in Basic Research (BBR-029), Youth Innovation Promotion Association CAS and Xplore Prize.
References
[1] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukeuoglu, Thore Graepel, and Demis Hassabis. Mas- tering the game of go with deep neural networks and tree search. Nat., 5297758)7444444448 2016.
[2] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. J. Mach. Learn. Res., 17:39:1-39:40, 2016.
[3] Jiř Kubalík, Eduard Alibekov, and Robert Babuška. Optimal control via reinforcement learning with symbolic policy approximation. JFAC-PupersOOnLine, 50(1):46224116, 2017. 20th IFAC World Congress.
[4] Daniel Hein, Steffen Udluft, and Thomas A. Runkler. Interpretable policies for reinforcement learning by genetic programming. Eng. Appl. Artif. Intell., 15:11666 2018.
[5] Mikel Landajuela Larma, Brenden K. Petersen, Sookyung Kim, Cláudio Santiago, Ruben Glatt, T. Nathan Mundhenk, Jacob F. Pettit, and Daniel Faissol. Discovering symbolic policies with deep reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5979-5989, PMLR, 2021.
[6] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. rl2: Fast reinforcement learning via slow reinforcement learning, arXiv preprint 161vv2661 2016.
[7] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.
[8] Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander Smola. Meta-q-learning. arXiv preprint arXiv: 1910.0015,, 201.
[9] Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What where, and how to adapt in transfer reinforcement learning. arXiv preprint arXiv:2107.02729 2021.
[10] Miles D. Cranmer, Alvaro Gonzale, Peter W. Battaglia, Rui Xu, Kyle Cranmer, David N. Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. CoRR, abs/2006.11287 2020.
[11) Shuyao Cheng, Pengwei Jin, Qi Guo, Zidong Du, Rui Zhang, Yunhao Tian, Xing Hu, Yongwei Zhao, Yifan Hao, Xiangtao Guan, Husheng Han, Zhengyue Zhao, Ximing Liu, Ling Li, Xishan Zhang, Yuęjie Chu, Weilong Mao, Tianshi Chen, and Yunji Chen. Pushing the limits of machine design: Automated epu design with ai, 2023.
[12] John R. Koza. Genetic programming the programming of computers by means of natural selection. Complex adaptive systems. MIT Press, 1993.
[13] Michael D. Schmidt and Hod Lipson. Age-fitness pareto optimization. In Martin Pelikan and Jürgen Branke, editors, Genetic and Evolutionary Computation Conference, GECCO 2010, Proceedings, Portland, Oregon, USA, July 7-11, 2010, pages 543-544. ACM, 2010.
11

---

[14] William G. La Cava, Tilak Raj Singh, James Taggart, Srinivas Suri, and Jason H. Moore. Learn- ing concise representations for regression by evolving networks trees, In 7th International Conference Learning Representations, ICLR 2019, New Orleans, IA, USA, May 6-9, 2019. OpenReview.net, 2019.
[15] Silviu-Marian Udrescu and Max Tegmark. Al feynman: Schysics-inspires method for symbolic regression. CoRR, abs/ abs//905.11 1481, 2019.
[16] Brenden K. Petersen, Mikel Landajuela Larma, T. Nathan Mundhenk, Cláudio Prata Santiago, Sookyung Kim, and Joanne Taery Kim. Deep symbolic regression: Recovering mathematical expressions from data via seeking policy gradients. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[17] Georg Martius and Christoph H Lampert. Extrapolation and learning equations. arXiv preprint arXiv:1610.02999 2016.
[18] Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems, 31, 2018.
[19] Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pages 5045-5054. PMLR, 2018.
[20] Jurgen Schmidhuber. Evolutionary principles in self-refcrential learning. on learning now to learn: The petta-met-meeta..-non Diploma thesis, Technische Universitat Munchen, Germany, 14 May 1987.
[21] Y. Bengio, S. Bengio, and J. Cloutier. Learning a synaptic learning rule. In JJCNN-9I-Seed -Seattle International Joint Conference Neural Networks, volume ii, pages 969 vol.2-, 1991.
[22] Sebastian Thrun and Lorien Pratt. Learning to learn. 1998.
[23] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint 1707vv/77..0980 2017.
[24] Kenny Young, Baoxiang Wang, and Matthew Taylor. Metatrace actor-critic: Online step- size tuning by meta grtt--grriint descent for reinforcement learning control. arXiv preprint 1805.00800.0554 2018.
[25] Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning learn: Meta-critic networks for sample efficient learning. arXiv preprint arXiv:1706.0952 2017.
[26] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
[27] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint arXiv: / 8870006844 2018
[28] Bradly Stadie, Ge Yang, Rein Houthooft, Peter Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. The importance of sampling iemetu-reinfoorcmment learning. Advances in Neural Information Processing Systems, 31, 2018.
[29] Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. In International Conference on Machine Learning, pages 9301 9312. PMLR, 2021.
[30] Haotian Fu, Hongyao Tang. Jianye Hao, Chen Chen, Xidong Feng, Dong Li, and Wulong Liu. Towards effective context for rentrrreitf learning: an approach based on contrastive learning, arXiv preprint arXiv:2009. 133891,00911381, 2020.
[31] Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and Chongjie Zhang. Metacure: Meta reinforcement learning with empoverment-driven exploration. In International Conference Machine Learning, pages 12600-12610. PMLR, 2021.
12

---

[32] Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep RL via meta-learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[33] Subham S. Sahoo, Christoph H. Lampert, and Georg Martius. Learning equations for extrap- olation and control. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4439 4447. PMLR, 2018
[34] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[35] Suraj Srinivas, Akshayvarun Subramanya, and R Venkatesh Babu. Training sparse neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 138-145, 2017.
[36] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint 172.011/ 2017.
[37] Xiao Zhou, Weizhong Zhang, Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3599-36000 2021.
[38] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: 1.00:///11000000 2016.
[39] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[40] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1666000 1606.01540, 2016.
[41] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016.
[42] John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Francis R. Bach and David M. Blei, editors, Proceedings the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1889-1897. JMLR.org, 2015.
[43] Volodymyr Mnih, Adrià Puigdomènech Badia, Mchdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep rein- forcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning. ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1928-1937. JMLR.org. 2016.
[44] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR,
2017.
[45] Yuhuai Wu, Elman Mansimov, Roger B. Grosse, Shun Liao, and Jimmy Ba. Scalable trust- region method for deep reinforeement learning using senecker-factored approximation. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017 Long Beach, CA, USA, pages 5279-5286 2017.
13

---

[46] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause, editors. Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmüssson Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1582-1591. PMLR, 2018.
[47] Antonin Raffin. RI baselines Zoo. sttra///gttaub.coo/rrrrfii///// 2018.
200
[48] F. Nogueira. Bayesian Optimization: Open source constrained global optimization tool for Python. URL https://github. condfinffuBaaeesiadpppiiiita
2014.
[49] Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.
[50] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[51] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style. high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[52] Tom Silver, Kelsey R. Allen, Alex K. Lew, Leslie Pack Kaclbling, and Josh Tenenbaum. Few- shot bayesian imitation learning with logical program policies. In The Thirty-Fourd AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty-eecond Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 10251-10258. AAAI Press, 2020.
[53] Abhinav Verma, Hoang Minh Le, Yisong Yue, and Swarat Chaudhuri. Imitation--poojcted programmatic reinforcement learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buc Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Sys- tems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 15726-15737, 2019.
[54] Dweep Trivedi, Jesse Zhang, Shao-Hua Sun, and Joseph J. Lim. Learning to synthesize programs interpretable and generalizable policies. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 25146-25163, 2021.
[55] Yichen Yang, Jeevana Priya Inala, Osbert Bastani, Yewen Pu, Armando Martin C. Rinard. Program synthesis guided reinforcement learning for partially observed environments. In Marc' Aurelio Ranzato, Alína Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 29669-29683, 2021.
and
[56] Wenjie Qiu and He Zhu. Programmatic reinforcement learning without oracles. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[57] Shao-Hua Sun, Te-Lin Wu, and Joseph J. Lim. Program guided agent. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Eahiopia, April 26-30, 2020. OpenReview.net, 2020.
14

---

[58] Andrew Howard, Ruoming Pang, Hartwig Adam, Quoe V. Le, Mark Sandler, Bo Chen, Weijun Wang. Liang- Chich Chen, Mingxing Tan, Grace Chu. Vijay Vasudevan, and Yukun Zhu. Search- ing for mobilenetv3. In 2019 IEEF/CVF International Conference Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 -Nvember 2, 2019, pages 1314-1324. IEFE, 2019.
[59] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[60] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. CoRR, abs/1 09050, 2018.
[61] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[62] Chenxi Liu, Liang-Chien Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 82-92. Computer Vision Foundation IEEE, 2019.
[63] Alexander L. Gaunt, Mare Brockschmidt, Rishabh Singh, Nate Kushman, Pushmeet Kohli, Jonathan Taylor, and Daniel Tarlow. Terpret: A probabilistic programming language for program induction. CoRR, abs/1608 (oos/660.04428, 2016.
[64] Matko Bosnjak, Tim Rocktäschel, Jason Naradowsky, and Sebastian Riedel. Programming with a differentiable forth interpreter. Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 547-556. PMLR, 2017.
[65] Bernhard Wymann, Eric Espié, Christophe Guionneau, Christos Dimitrakakis, Rémi Coulom, and Andrew Sumner. Tores, the open racing car simulator. Software available at http://torcs. sourceforge. net, 4(6):2, 2000.
15

---

A Symbolic Operators
To ensure the numerical stability of the proposed symbolic learning framework, we regularize the operators and employ a penalty term to keep the input from the "forbidden" area. We show the operators and the corresponding penalty terms as follows:
Multiplying operator:
min(maz(21 - 100), 100) min(max(22, -100), 100),
the penalty term can be formulated as:
Linat = max [1 - 100,
11, -) - 100, maar
Division operator:
0, 22 < 0.01,
22 2 0.01.
the penalty term can be formulated as:
Ldiv (0.00 I2, 0)
Sine operator: y = sin(r), the penalty term is set as zero: Ligin = 0.
Cosine operator: y cos(2). the penalty term as zero: Leos 0.
Exponential operator:
= wawpmim((maraa.-aa 4)),
the penalty term can formulated as:
Leap maxl( 4,0) mmar(-10- x. 0)
Log operator:
log(mar(1,, 0.001)),
the penalty term can be formulated
Ltog = max x, 0)
Identity operators: I, the penalty term is set zero: Lidentity 0.
Condition operator: = sigmoid(z1) * 22 + (1 sigmoid(z:)) * æ3. the penalty ter is
set zero: Leondition O.
In practice, the identity operator is only used in the plain structure. For all the environments in single- task RL and meta-RL, we use the symbolic network structure described in Section 3.1. Especially, for the meta-RL environment Cheetah-vel-odd think the Condition operator will be useful for environments where the reward function changes. During training, we involve a penalty loss function Cpenalty which is the sum of the penalty terms of regularized operators in symbolic networks:
we add one Condition operator each layer. We
i=
Epenalty
Cg. (2x))
(6)
We show the learning curves of the penalty loss function single-task RL in Figure 5 and for meta-RL.n in Figure During the training process, for all environments in both single-task RI. and meta-RI,, the penalty loss function remains on a very small order of magnitude, which indicates that most of the operators in the symbolic network work the same the original unregularized operators.
16

---

CartPole
MountainCar
Pendulum
InvooublePend
ponalty Loos
penalty
eTW steps
emv steps LunarLander
env steps
cni steps EipedaWWelker
InvPencSuinuup
penally 1095
penalty toss
scnalty 1054
ponalty 1002
Figure 5: Learning curves of the penalty loss function single-task RL. The shaded area spans one standard deviation.
Hopper-paree
Wa Walker2d params
Cheetah-vel-ood
penalty 105s
penally loss
pena ity
CTV steps
env sitpps
Cni steps
Lunarlander paramess---amssssss
InvDoublePend pard-aasmmsdssss
Cartpole-fioo
penalty loss
penalty losss
8.001 023m
1:0M eTiv steps
1201
L10%
2.cm
2205
1.00
steps
steps
Figure 6: Learning curves of the penalty loss function in meta-RL.
B Environment details.
B.1
Single-task RL
For single-task RL, we use the same benchmark control tasks as DSP 5 We describe these environments as follows:
CartPole: In the environment, there is cart that move linearly with a pole fixed on it.
The goal is to balance the pole by applying forces in the left and right direction on the cart. The max horizon length is set 1000. We continuous version from an open-source implementation httpt::t///ee //gist tigt..u..... com/ / andanforth
MountainCar: In the environment, there is placed stochastically at the bottom of sinusoidal valley. The action is the accelerating the car in either direction. The goal of the environment is accelerate the reach the goal state on top of the right hill. We use CountainCarrontiiuuuuu- from OpenAl Gym
Pendulum: In this environment, there is pendulum attached to fixed point at one end. The pendulum starts in a random position and the goal is to apply torque on the free end to swing into upright position. We use Pendulum-vO, from OpenAl Gym.
17

---

Algorithm The training process of ESPL.
Input: The number of iterations for temperature and Lo norm schedule f, The target temperature t and target minimum Lg norm 4. The symbolic network S.V and the parameters w, b. The probabilities p of the path selector.
Initialize replay buffers B.
2: for training iteration f = to T - 1 do
3: T (1 (1-7) T3) (1 min(t,())) +
fo 4: Generate symbolic policy Zgm with the symbolic network and the path selector.
5: for environment step & = 0 K do
6: Collect data with ~ NN (s), F())) and add to buffer B end for
7:
8: Imin =4 + (1
mii((1.))) 1,
2
9:
for steps in training step do
Sample RI. Batch from the buffer Sample with gumbel sigmoid and ST trick. Obtain the policy: (ww)
11:
12:
13:
Calculate loss for the Critic: Leritic Case
14:
Calculate loss for the Actor:
Lactor end for
Lase + a. Epenatty + ar2Lselect
15:
Update the 02 with Leritic
17:
Update w, b. p. Op with Lactor
18:
end for
InvertedDoubleeendulmn In this environment, there is cart that can move linearly, with pole fixed on it and second pole fixed on the other end of the first one. The cart can be pushed left or right, and the goal is balance the second pole on top of the first pole by applying continuous forces on the cart. We use BuirreeDDDubeeeedduuuBulleeeee PyBullet [49]
from
This environment a combination of CartPole and Pendulum. The goal of the environment is to swing-up the pendulum from its natural pendent position its inverted position. We use
from PyBullet.
LunarLander: In this environment, there is spacecraft with a main engine and two orienta- tion engines. The goal of the environment is to land spacecraft on a landing pad without from OpenAl Gym.
crashing by controlling the engines. We use LunarLanddeCConimmouo
Hopper: In this environment, there is two-dimensional one-legged robot that should move forward. We use HopperBulletEnn-v0 from PyBullet.
Bipkda/Walker: This is a simple 4-joint walker robot environment. The goal is to control the walker move forward. We use BipedalWalker-v2 from OpenAl Gym.
B.2 Meta-RI.
We introduce the details of the enviromments used in the meta-RL, RL experiments. We build all the meta-RI. environments by modifying the origin environments of OpenAl Gyn We describe these environments as follows:
Walker2d -aramms:---mm::: This environment is built by modifying the OpenAl gym Walker2d, in which two-dimensional two-legged robot should move forward. For each task, the dynamics parameters of the environment are randomly initialized. The horizon length is set 200. For the experiment, sample 40 training tasks and 10 test tasks.
Cheetah-vel-ood This environment is built by modifying the OpenAl gym Half Cheetah. In the modified environment, a 2-dimensional robot with nine links should achieve target
'We build our meta-RL RL https//giihubbccmm/memmms/////an Bareoddd__aa_Peee envs.
environments
based
on
the random- param-ent
18

---

Algorithm 2 The training process of CSP.
Batch of training tasks fi= The number of iterations of temperature and target Lo norm schedule i.. Target temperature тτ. The target Lo norm of the mask at the end of training 4,- The scale of the penalty loss a1. The scale of the loss to regularize the sum of probabilities oz. The scale of the loss for the KL divergence B.
Input:
1: Initialize replay buffers B, for each training tasks.
2: for training iteration 0 to
do
3: T T= (1) (1 mén(t.t.) +
4: for each task K; do
5: Initialize context C 0
6: for k = to K do
7:
Sample 2 ~ q0, (2/cz)
8:
Generate symbolic policy Taym with (22) and (2,
9:
Collect data with a ~ N (Tegm (8), F(8,21 =)) and add to buffer B,. Update c; [(5), Og s'y Tyy f NN B,
11:
end for
end for
13:
for steps in training step do
Sample batch of tasks. for each R; in the batch do
15:
Sample context 6 and RL Batch b, from the buffer B,
17:
Sample context variables
qo, (=le;)
18:
Obtain parameters w, b with (2))
Obtain probabilities p with (z)
20:
Sample m with gumbel sigmoid and ST trick. Obtain the policy: SNWw))
hearget- = 4 + (1 (1- * (1- min((..s)
22:
23:
Calculate loss for the critic: Cioittic Leee
critic
Calculate loss for the Actor:
Cactor Lase + an peennatyy + oz @12huunleet
25:
Calculate the KL. divergence L'KI. = (21 1)) end for
27:
Update the Critic with ΣiEr Coratic
28:
Update the context encoder with E + BBkk))
29:
Update the path collector and the parameter generator with Eizr Sactor
30:
end for
31:
end for
velocity running forward. For training task, the target velocity is sampled uniformly from [0, 2.5]. For test task, the target velocity is sampled uniformly from [2.5, 3.0]. The horizon length is set as 200. For the experiment, we sample 50 training tasks and 15 test tasks.
Hopper-parane This environment is built by modifying the OpenAl gym Hopper, in which a two-dimensional one-legged robot should move forward. For each task, the dynamics parameters of the environment are randomly initialized. The horizon length is set as 200. For the experiment, we sample 40 training tasks and 10 test tasks.
samarllander-parmmt This environment is built by modifying the OpenAl gym Lunar Lander, in which rocket should land on the landing pad. For each task, we randomly initialize the gravity. The horizon length is set as 1000. For the experiment, we sample 40 training tasks and 10 test tasks.
In-DoublePend-pareee params: This environment is built by modifying the OpenAl gym Inverted Double Pendulum. this environment, there a cart that can move linearly. pole fixed on it, and second pole is fixed on the other end of the first pole. For each task, the dynamics parameters of the environment are randomly initialized. The horizon length is set 200. For the experiment, we sample 40 training tasks and 10 test tasks.
19

---

Table 7: Measure of uncertainty for ESPL.
Environment
Measure of uncertainty
CartPole
0.4998
MountainCar
0,4998
Pendulum
0.4997
Inv DoublePend
0.4995
Inv InvPendSwingup
0,4999
LunarLander
0.4997
Hopper
0.4999
BipedalWand Walker
0.4998
Cartpole-ft-ood:
This environment built by modifying the OpenAl gym Cart Pole In the environment, there is cart that move linearly with pole fixed on it. For each task, we randomly initialize the force magnitude and the length of the pole. For a training task, the force magnitude and the length of the pole sampled uniformly from [7.5, 12.5) and [0.3, 0.7]. For a test task, the force magnitude and the length the pole is sampled uniformly from [5, 7.5] U [12.5, 15] and [0.2, 0.3] [0.7, 0.8). The horizon length is set as 200. For the experiment, we sample 40 training tasks and test tasks.
For all the environments of single-task RL and meta-RL, we wrap the action interface with the tanh function limit the range of the action.
C Implementation Details
C.1 Single-task RI.
During training, we employ two neural networks to approximate the Q function as the original SAC. The neural networks have two hidden layers with 256 hidden units. Besides, to limit the probabilities in the range of [0, 1]. we clip the probabilities after each update. After training, even if the mask is sampled from the Bernoulli distribution with probabilities p. the process of obtaining the symbolic policy is almost deterministic, because empirical observations show that almost all probabilities converge to or after training. We use a measure of uncertainty which the average difference between probability p and 0.5. When the measure approaches 0.5, the process of obtaining the symbolic policy almost deterministic, We report the measure Table 7 For the details of training process, give the pseudo-code in Algorithm
C.2 Meta-RL
In practice, we build up our off-policy learning framework on top of the soft actor-critic algorithm (39 following PEARL. To construct stochastic policy, also employ small neural network F to output the standard deviation just like single-task RL. The neural network has two hidden layers with 64 hidden units. Note that this neural network is only used during training. During the evaluation. we only use the produced symbolic policy to infer the action. Besides, to limit the probabilities produced by (=) in the range of (0, 1), employ the sigmoid function. We initialize the bias of the last layer in (2) (2) as 3.0. As sigmoid(3.0) = 0.9526, we will have most of the paths of the symbolic network active the beginning of training and ensure that the input of the sigmoid function is not too large to prevent the gradient from disappearing during training. For the details of
the training process, we give the pseudo-code in Algorithm 2
'We use the same continuous version single-task CartPole.
20

---

Table 8: Hyperparameters for the ESPL.
Parameter
Value
optimizer
Adam (50
number of samples per minibatch
256
scale of the reward
1
learning rate discount
3 10
0.99
sac target smoothing coefficient
0.005
target temperature
0.2
training steps per iteration scale of the penalty loss
1000
1
Table 9: Environment Specific Hyperparameters for ESPL.
Environment
Scale of Leelect
Target lo norm ratio
Schedule iterations
CartPole
0.08
0.002
400
MountainCar
0.64
0.002
200
Pendulum
0.08
0.002
300
DoubleePdd
0.08
0.005
400
InvPendS wingup
0.1
0.005
400
LunarLander
0.08
0.005
600
Hopper
0.08
0.01
2000
Bipedal Walker
0.64
0.01
1400
D Experiment details
D.1 Single-task RL
In this section, we give the main hyperparameters ESPL for single-task RL. We show the common hyperparameters of ESPL in Table We also list the environment specific hyperparameters in Table For the scale of Lselect, we choose the best one from {0.04, 0.08, 0.1, 0.16, 0.32, 0.64]. The target To norm ratio is the ratio of the target Lo norm of the mask the end of training l. to the number of parameters the symbolic network. We set the value according to the complexity of the task and do not tune the value. We show the learning curves of the average Lo norm ratio of the mask for the sampled tasks in Figure 8 The Lo norm ratio at the end of training is always higher than the target lo norm ratio. Thus, the Lo norm ratio at the end of training is more affected by the scale of Leetect. The schedule iterations mean the number of iterations temperature and target Lo norm schedule. Then illustrate the evaluation details about symbolic policies. DSP performs independent training runs with different random seeds for each dimension the action, selecting the best symbolic policy the end of training. Thus, for environments with n-dimension action, they perform 3n training
9
Table 10: Hyperparameters for the CSP.
Parameter
Value
optimizer
Adam 50
number of samples per minibatch
256
scale of the reward
5
learning rate scale of the kl divergence loss
3
10
I
discount
0.99
sac target smoothing coefficient
0.005
target temperature
0.2
training steps per iteration
2000
scale of the penalty loss
1
21

---

Table 11: Environment Specific Hyperparameters for CSP.
Environment
Meta batchsize
Scale of Leelect
Target lo norm ratio
Schedule iterations
Walker2d-params
10
0.25
0,01
450
Hopper-params
0.25
0,01
300
10
2.0
0,01
150
Cartpole-fl-os floo
0.25
0,002
25
Lunarlander-g
10
0.25
0,01
60
Cheetah-vel vel-ood
2.0
0,01
300
MountaisCar
Pendiilum
InDonsbleterd
ret0
0 0.004
0,002L
0,218
10 19/0
10 200
stegs
oTT steps
Figure 7: Learning curves of the average Lo norm ratio of the mask for single-task RI..
runs and select the single best policy. The symbolic regression method repeats each experiment for single dimension of action 10 times using different random seeds and selects the expression with the lowest mean-square error. For all the environments, the proposed ESPL performs 3 independent training runs and selects the single best policy.
D.2 Meta-RL
In this section, give the main hyperparameters of CSP for meta-RL. We show the common hyperparameters of CSP in Table TO We also list the environment specific hyperparameters in Table TT The meta batchsize is the number of sampled tasks per training step. We set it cording to the number of training tasks. For the scale of Leelect, we choose the best one from {0.1, 0.15, 0.2, 0.25, 0.5, 1.0, 2.03. The target lo norm ratio is the ratio of the target Lo norm of the mask at the end of training Le to the number of parameters of the symbolic network. We set the value according the complexity of the task and not tune the value. We show the learning curves of the average Lu norm ratio of the mask for the sampled tasks in Figure 8 The Lu norm ratio at the end of training is always higher than the target lo norm ratio. Thus, the Lo norm ratio the end of training is more affected by the scale of Lsetect- The schedule iterations mean the number of iterations of temperature and target Lo norm schedule. end the schedule near the end of training but for Cartpole-fl-ood in which the policy converges quickly, we reduce the number of schedule
Table 12; Average count of all selected paths and paths selected by at least ninety percent policies.
Environment
Selected paths
Mostly selected paths
Walker2d-params
76.42
70.3
Hopper-params
21.5
20.33
Gooubbleendd-prams
23.2
21.0
Cartpole-floow flod
3.06
3.0
Lunarlander-g
5.2
5.0
Chectah-vel-oor
27.04
18.5
22

---

Hopper--arrmm
We Waaremsdd-parmms
Cheetah-vel-ood
0,038
0.001
0.023
CnV Stops
env steps
env steps
Lunariander-pon params
InvDoubleeend- Parrppbleendd-params parapaaaaas
Cartpole-f-aod
0.017
0.028
0.009
steps
steps
steps
Figure 8: Learning curves of the average Lo norm ratio of the mask for the sampled tasks in Meta-RL.
iterations. We run all the experiments with five random seeds and average the results to plot the learning curve.
D.3 Platform and Device
The implementation of our ESPL and CSP is based the p51 Besides, train the proposed CSP and ESPL with Nvidia V100 GPU. When evaluating the inference time, we use Intel(R) Xeon(R) Gold 5218R @ 2.10GHz CPU.
E Extended Related Work
In this section, we provide an extended review of related work.
E.1 Programmatic RL and program-guideds RL
Programmatic RL represents policies as short programs to improve the interpretability of the policies. A series of works[18] (2
19 53 54) obtain the programmatic policies by distilling them from neural network policies. VIPER [ extracts neural network policies into a decision tree by imitation learning algorithm Q Q-DAGGER to get verifiable policies. LPP(52) also employ imitation learning but exploit the logical structure and use domain-specific language to avoid enginecring each individual feature. [DPS[19] optimize the program by imitation learning from neural network policy and employ an instance-specific syntactic model called sketches as guidance. PROPEL (53] views the unconstrained policy space as mixing neural and programmatic representations and uses mirror descent to optimize the policy. They evaluate their method in continuous control domains and show better results. LEAPS[54) (54) learns the program embedding from given programs and uses CEM to search latent programs. MPPS (55) uses conditional variational auto-encoder and program synthesis to solve the partially observed problem and obtain the policy with MaxSAT. PRL[56) learns the programmatic policies by introducing differentiable program derivation tree and searching the structure of the tree with gradient. Although programmatic RL methods also aim to obtain simple and interpretable policies, the ESPL proposed in this paper represents policies as symbolic or algebraic expressions which are different from programmatic RL. The different forms of symbolic expressions and programs lead to different difficulties in obtaining policies. This paper proposes a solution to the low data efficiency of the existing symbolic policy methods and extends it to reinforeifrr learning. The scope of the application is also different. Some programmatic reinforcement learning [4 (55] interacts with the environment through DSL and is not applicable to continuous control tasks. In contrast, symbolic policy works often evaluate their methods in continuous control tasks, e.g. DSP. In addition, to further verify our method, we compared the proposed ESPL with some programmatic RL methods. Puodram-guided [L|57] trains the agent to
23

---

432
1.4
retur
retum
/
0.2
214
110
CSP _
CSP D
CSP L1 CSP
CSP
1.094
1330
3354
3350
3.20
env steps
env steps
Figure 9: Ablation results of the symbolic network structure (left), the path selector(right). CSP P means the plain structure and CSP D means the densely connected structure. CSP LI means replacing the path selector with the L1 norm minimization.
accomplish the tasks specified by a program and use a multitask policy to accomplish subtasks of the overall task. This approach is also interpretable but dependent on given program, rather than learning an interpretable policy.
E.2 Neural Architecture Search
Neural Architecture Search (NAS) 59 60 aims at automatically designing a neural network for certain task and dataset and has been promising approach to automate deep learning applications. DARTs[60] is one of the representative works of gradient-based neural architecture search [60] (2). These works formulate super-network based the continuous relaxation of the architecture representation, which allows efficient search of the architecture using gradient descent. Although we share the similar motivation efficiently search by gradient descent and both have the process of selecting the appropriate substructure from a large structure, the algorithms are still quite different. These works use the softmax function select one proper neural network operation from set of candidate operations for each connection. In contrast, we learn to mask out redundant connections with the proposed path selector and only very small percentage of connections are selected (See Figure In addition, they usually divide the computer vision dataset and alternate training structures and parameters, whereas we use completely different training approach. Besides, these works aim to search for good neural network structure for computer vision while we aim to find compact symbolic policies for reinforcement learning. We also involve meta-RL. and develop contextual symbolic policies which can adapt to new tasks, while they usually need finetune or re-training for new tasks.
Building differentiable frameworks and using gradient-based methods to obtain solutions have indeed found applications in many fields, e.g. neural architecture search, and program induction [63] 64). However, compared to ESPL, these works differ significantly in terms of (1) the constructed differentiable frameworks, (2) the process of obtaining solutions through gradients, and (3) the specific problem domains they target.
More Experiment Results
F.1 Comparison with Programmatical RL
In the main body of this paper, considered that the form of the policies will affect the final performance. Thus, except for neural network policies, we selected Regression and DSP, which also represent policies with symbolic expressions for comparison order ensure the faimess of comparison. We would like to compare the proposed ESPL with some programmatical RL methods to further verify method in this section. We carry out experiments The Open Racing Car Simulator (TORCS) which provides a rich environment with input from up to 89 sensors.
We first follow the experiment settings in NDPS 19, where the controller tasks the input from 29 sensors and decides values for the acceleration, and steering. We show the lap time seconds (lower is better) in Table I3

---

Table 13: Comparison results in TORCS following experiment settings in NDPS.
CG-SPEEDWAY-I
AALBORG
ESPL
50.23
107.3
NDPS
61.56
158.87
The results demonstrate that the symbolic policies produced by ESPL achieve higher performance. NDPS learns the programmatical policies from neural network policy and the object mismatch problem may hurt the performance. We also provide the example video in https: / anonymous 4eoee..sciieeeeeee science/r/ ESPL- 8F5B
We then follow the experiment settings in PROPEL where the controller tasks the input from 29 sensors and decides values for the acceleration, steering, and brake. We carry out experiments in three maps and show the lap time in seconds (lower is better) in Table [4] In this table. VIPER[[8] method that extracts neural network policies into decision tree by imitation learning. The PROIR means the PID controller that is also used as the initial policy in PROPEL. PROPELTREE means PROPEL using regression trees. PROPELPROG means PROPEL using a high-level language similar to NDPS. In these three maps, the symbolic policies produced by ESPL outperform other programmatical polcies and is comparable with PROPELTREE. The proposed ESPL train the symbolic policies from scratch without initial policies which are used in PROPEL. We also compare
Table 14: Comparison results in TORCS following experiment settings in PROPEL.
G-Track
E-ROAD
AALBORG
ESPL VIPER I8
77.31
79.67
107.94
83.60
87.53
110.57
PRIOR
312.92
322.59
244.19
N9
108.25
126.80
163.25
PROPELPROG 53
93.67
119.17
147.28
PROPELTREE[53]
78.33
79.39
109.83
the proposed ESPL with PRL[56] on continuous control benchmarks presented in DSP in Table PRL represents the policy with program derivation tree. They use affine policy which linear transformation as domain-specife programming language (DSL) for Gym environments. The proposed ESPL achieve higher performance in most environments.
15
Table 15: Comparison with PRL continuous control benchmarks presented in DSP.
CartPole
MountainCar
Pendulum
Dnvbleeleeend
PendSSiiiggupp
LunarLander
Hopper
BipedalWalker
ESPL
1000
94.02
-151.72 -17.566
9359.9 9357.93
890.36 889.47
283.56
2442.48
309.43
PRL 56
1000
95.07
265.17
2226.49
270.33
F22
Ablation Study for Meta-RI
In this section, we carry out experiments by ablating the features of CSP. We change the structure of the symbolic network and replaced the path selector with the L1 norm minimization. we compare the learning curves of the test task rewards in Figure The results show that the dense connections effectively improve the performance and arranging the operators to further improve the performance. Besides, the path selector is better at selecting the symbolic policy from the symbolic network than L₁ norm minimization.
facilitate the search for the proper symbolic form by
F.3 Human Study for Interpretability
We also carry out human study to evaluate the interpretability of the symbolic policies generated by ESPL. The assessment of policy interpretability actually requires some understanding of the environments and what each state variable means. We invited ten researchers to rate the interpretability of policies, with maximum of 20 minutes for each policy. On a five-point scale, we told them that
25

---

the interpretability could be judged based on whether they could see from the policy expression what the policies was based on make decisions and the possible corresponding relationships between actions and states. A score of five indicated that the strategy was highly interpretable and could be designed by humans, while a score of zero indicated that it was completely uninterpretable, just like the neural network policies. We measured the average score of the interpretability of the policies. We give the interpretability scores for ESPL and DSP in Table I6
Table 16: Comparasion of interpretability scores of ESPL and DSP.
CartPole MountainCar Pendulum
InvDoublePend
Inv PendSSwiingu
Landrr
Hopper
BipedalWalker
ESPL
4.5
4.1
4.3
3.9
2.6
3.1
DSP
4.5
5.0
4.2
3.1
32
Based on the results of the human study, we found that when the symbolic expressions were relatively short, there was consensus that the expressions were highly interpretable. When the expression length became longer, the interpretability score decreased. It may be relatively difficult to understand symbolic expressions in short time, but it is generally believed that interpretability is much higher than a black box, at least they can understand the action will be affected by which state, and the partial correlation. Besides, the symbolie policies produced by ESPL and DSP have comparable interpretability.
F.4 Analysis of symbolic policies for Meta-RL
We then analyze the symbolic policies for different tasks produced by CSP. For each environment, sample 10 tasks from the environment task distribution and obtain the corresponding symbolic policies with CSP. Then analyze the selected paths of these policies which determinc the forms of the symbolic expressions. Table shows the results. We calculate the average count of selected paths per action dimension among the policies We find that this number varies across different environments. The symbolic expression can be extremely short for simple environments like Cartpole or relatively long for complex environments like Walker2D. We also calculate the average count of paths that are selected by more than ninety percent of the symbolic policies. In almost all environments, the mostly selected paths account for high percentage of the selected paths, which indicates that the expressions of symbolic policies for different tasks of the same environment share similar forms.
F.5 Theoretical Analysis for Cartpole
The theoretical analysis is as follows: The dynamic of cartpole system can be defined as:
8fa + 2m sin 0 ( 3g cos f)
8M 3m cos 20 + 5mn
0 g sin f cos 0 (fa Lmdd sin
n M
122 cos 8
VTL + M
Where f is the coefficient of force, a represents the action, m is the weight of the pole, M the weight of the cart, L is the half-pole length, 0 is the angle between the pole and the vertical direction, and æ denotes the horizontal coordinate of the cart.
8 2m sin 6' 3g oos
8MM 3m 5m
Define X
then
9 8 of 22
L
2h 2
'We only consider paths that contribute to the final expression. Besides, the count of the remaining paths may not equal to the count of operators in the final symbolic policy because some operators can be merged after simplifying symbolic expressions.
26

---

According to the HartmannGGrobman theorem, the local stability of this nonlinear system near its equilibrium point equivalent the linearized system near the equilibrium point. For cartpole system, the equilibrium point is [x. à 0, 6) [0. 0, 0, 0)
If a O, the system can be linearized as:
0
6am
0
X
8M 2m
0
0
0
0
LC
Calculate its eigenvalues:
[0, 0, 3.971145993, -3.97114598)
Due the presence of positive eigenvalues, according the Hartman-GGrobman theorem, the system unstable.
If a 17.170 + 1.20, linearize the system near the equilibrium point:
0
0
0
0
137.36f
137
9.6f
X
0
0 0
0
1.2/ L M 33
L1
m
ml
Calculate its eigenvalues:
00 + 0.j 26.34 0, -2122666.26662. 26.34 6.65014286j
Since all the real parts of the eigenvalues are non-positive, according to the Hartman Hartman-Grobman theorem, the system is stable. Therefore, for the CartPole environment, the policies learned through ESPL can maintain the stability of the CartPole system.
27


Reviews:
[{"review_id": "Whn65tjUSe", "reviewer": "Reviewer_iXeo", "summary": "The paper proposes ESPL, a method for learning symbolic policies in environments with low-dimensional state spaces. ESPL uses a densely connected neural network structure (like DenseNet), where the activations in each layer are replaced with a hand-picked set of functions, such as multiplication, division, log and and exp. Second, ESPL uses the Gumbel-Softmax trick to learn a masking function with a minimal $L_0$ norm. This function ensures that the method learns compact symbolic policies by masking out redundant pathways in the neural network. ESPL is shown to be 100 to 1000 times more sample-efficient than a prior symbolic learning approach in single-task RL. The authors also propose a version that can be conditioned on task context for meta-RL.", "questions": "1. Is Figure 1 right the exact architecture used in the experiments? Or do you hand-design a different arrangement of activations for each experiment?\n\n2. Is it possible to scale symbolic regression to high-dimensional state spaces?", "limitations": "The limitations of ESPL are not explicitly addressed. I am especially curious about the possibly of scaling up to high-dimensional (e.g. image) state spaces.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The proposed method is well motivated and clearly explained. $L_0$ regularization of the parameters of Bernoulli random variables is an elegant approach to learning compact symbolic expressions.\n2. In single-task RL, ESPL outperforms a prior symbolic learning approach $DSP^0$ with between 100 and 1000 fewer training samples.\n3. A task-conditioned version of ESPL called CSP outperforms prior (non-symbolic) approaches in a meta-learning setup. The authors point out that the symbolic policies are much faster to run than the baseline neural net policies.\n4. Source code is included.", "weaknesses": "1. The discovered symbolic policies in Table 1 seem to be somewhat more complex than the policies from $DSP^0$. It could be useful to additionally measure the complexity of the discovered policies for the various methods (if such measure exists).\n2. It is not very clear how much effort went into hand-picking the activation functions in each layer of the network. Is it necessary to hand-design a different network for each experiment / environment?\n\nMinor:\n* Missing space before `(` and `[`. This is repeated at least 10 times throughout the paper, e.g. lines 33 and 34.\n* Line 109: repeat what ESPL stands for.\n* Line 210: mixing up $t$ and $T$.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive review, insightful feedback and constructive comments that help improve the quality of the paper! We are glad to answer your questions and would appreciate any further response.\n\n> **Q1.** The discovered symbolic policies in Table 1 seem to be somewhat more complex than the policies from DSP. It could be useful to additionally measure the complexity of the discovered policies for the various methods (if such measure exists).\n\nA1. Thanks for your advice. \nBecause there was no well-defined measure in the previous work, we compare the length of the symbolic policies and define:\n$length=\\\\frac{\\\\sum^{i=n}_{i=1}N_o+N_c+N_v}{n}$, where n is the dimension of the action, $N_o$ is the number of operators, $N_c$ is the number of constant terms, $N$ is the number of variable terms. We give a comparison of the length of the symbolic policies in the following table.\n\n|  | Average | CartPole | MountainCar | Pendulum | InvDoublePend | InvPendSwingup | LunarLander | Hopper | BipedalWalker |\n|---|---|---|---|---|---|---|---|---|---|\n| ESPL | 12.91 | 3 | 6 | 7 | 15 | 13 | 16.5 | 24.6 | 17 |\n| DSP | 8.25 | 3 | 4 | 8 | 1 | 19 | 6.5 | 12 | 12.5 |\n\nIn the benchmark environments used in the literature, in some environments ESPL produces longer symbolic policies than DSP, in others ESPL produces similar or shorter symbolic policies than DSP. In general, symbolic policies produced by ESPL are only slightly longer than DSP's, and this degree of difference has little effect for the symbolic policies (with similar interpretability in global rebuttal 2). \n\n> **Q2.** It is not very clear how much effort went into hand-picking the activation functions in each layer of the network. Is it necessary to hand-design a different network for each experiment / environment?\n\nA2. We did not hand-design a different network for each environment. We use the same symbolic network structure for all environments (as stated in lines 154-155), and use the same set of symbolic operators as the DSP for a fair comparison.\n\n> **Q3.** Is Figure 1 right the exact architecture used in the experiments? Or do you hand-design a different arrangement of activations for each experiment?\n\nA3. Yes, we use the same architecture shown in Figure 1 right for all environments. We don't need to hand-design a different arrangement of activations for each environment.\n\n> **Q4.** The limitations of ESPL are not explicitly addressed. I am especially curious about the possibility of scaling up to high-dimensional (e.g. image) state spaces. \n\nA4. ESPL is currently designed for continuous action space environments with state vector inputs, but not for environments with discrete action space or high-dimensional observation inputs. As described in Appendix G, \nfor tasks with high-dimensional observation like images, the proposed ESPL and CSP can not directly generate a symbolic policy. But we can employ a neural network to extract the environmental variables and generate symbolic policy based on these environmental variables. Methods such as beta-VAE [1] or Slot Attention [2] can be used to extract environmental variables from high-dimensional observations.  We leave this in future work.\n\n[1] Higgins, Irina, et al. \"beta-vae: Learning basic visual concepts with a constrained variational framework.\" International conference on learning representations. 2016.  \n[2] Locatello, Francesco, et al. \"Object-centric learning with slot attention.\" Advances in Neural Information Processing Systems 33 (2020): 11525-11538.  \n\n>**Q5.** Is it possible to scale symbolic regression to high-dimensional state spaces\n\nPlease refer to A4.\n\nThanks aging for your detailed review. We will revise the symbols, expressions and formatting errors in the paper in revision.\n"}, {"review_id": "Jc5MCczk6H", "reviewer": "Reviewer_uiuA", "summary": "The paper \"Efficient Symbolic Polmicy Learning via Gradient Descent\" proposes a new neural symbolic architecture for agents learned via reinforcement learning. Authors propose an architecture with an alternance of symbolic and linear layers. To finally obtain simple expressions, a probabilistic mask is learned altogether with parameters, enabling the production of sparse archiectures resulting in small expressions. The approach is experimented on classical RL settings, and also on meta-rl ones. ", "questions": "- Is the main contribution of the paper the archiecture or its application to RL and meta-RL ? what about applying it on supervised learning tasks ? \n- How is q defined ? is it a deterministic output from a lstm or a gaussian from a transformer or ...? How is it trained ? \n- How are chosen the sequence of symbolic operators used in the architecture and what are they in the experiments ? how many (please specify m and n)\n- From my understanding, since we have m+2n inputs in the symbolic layer and only m+n outputs, the architecture has a with that reduces with layers. is it true ? \n- What is the impact of lmin in (5) ? Do you have experiments to analyze it ? \n- The metaRL objective given in the end of the first paragraph of page 6 is difficult to understand: the context is both sampled from the policy and used by the policy. Could you give clearer insights about this strange cycling definition ? Also, I would have expect that there is for each new task an exploration phase that would not been considered in the quantity to maximize : for k episodes no rewards, only collecting experience, such that we are good for the k+1th one in expectation depending on contexts from the k first episodes. Wouldn't it be better to present it like this ? \n\n  \n\n", "limitations": ".", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Very interesting architecture\n- Proposals look innovative\n- Clever way to obtain sparsity\n- Convincing experiments about efficiency ", "weaknesses": "- I cannot understand, if the architecture is new, why authors did not experiment it on classical supervised learning (eg, on classification tasks) before such much more reinforcement learning setting. Is the architecture only new in the field of RL ? Is the main contribution of the paper the archiecture or its application to RL and meta-RL ?\n- I feel that lacks important details about the used archiectures (for instances q is not defined, nothing is said about the choice of symbolic operators, nor about their numbers n and m, etc.)\n- Not much results about explanability, which is one of the main goal of the proposal", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments. We would like to clarify the concerns as follows:\n\n>**Q1.** I cannot understand, if the architecture is new ...\n\nA1. (1) This paper aims to propose an efficient and effective method for learning symbolic policies. Symbolic policies are designed for sequential decision tasks, which are typically addressed using reinforcement learning methods rather than supervised learning. \n(2) This approach is innovative and not a direct application of existing method of reinforcement learning. \nHowever, it cannot be directly applied to supervised learning tasks. Supervised learning tasks (e.g. image classification, machine translation) have very different input and output forms compared to sequential decision-making tasks. As for the symbolic network, we drew inspiration from prior works (line 116) in the field of symbolic regression. However, our symbolic network architecture, constraints on symbolic operations, training process, and the process of obtaining compact expressions are distinct. These methods cannot be directly applied to learning symbolic policies.\n(3) The contributions of this paper are three-fold. First, we introduce a novel gradient-based symbolic policy learning algorithm named ESPL that efficiently learns symbolic policies from scratch. The novelty is reflected in new symbolic network architecture, regularized operators, probabilistic path selectors, new loss functions, and training procedures, among others. Next, with ESPL, we develop the contextual symbolic policy for meta-RL, enabling the generation of symbolic policies for unseen tasks. This is a capability not achieved by any previous symbolic policy learning methods. Finally, we summarize our empirical results, which demonstrate the advantages of ESPL in both single-task RL and meta-RL scenarios.\n\n>**Q2.** I feel that lacks important details about the used ...\n\nA2. The Q-function (also known as the state-action-value function) is a crucial function in reinforcement learning, serving as a means to assess the expected return of different actions taken in a certain state. The Q-function is also referred to as the critic, and we update its parameters using the SAC algorithm, with pseudocode provided in Appendix C. The selection of symbolic operators is detailed in Section 3. To ensure a fair comparison, we adopt the same set of symbolic operations as in DSP. The architecture of the symbolic network used in experiments follows the structure shown on the right side of Figure 1 (lines 154-155). The values of m and n can also be derived from the network structure. For instance, if the next-layer symbolic operations are $[exp, log, mul, div]$, then m=2 and n=2, where m represents the number of unary functions and n represents the number of binary functions (line 123). \n\n\n>**Q3.** Not much results about explainability ...\n\nA3. (1) We give examples of interpretable analysis in Section 5.4 and report the symbolic policy lessons learned in Table 1. The symbolic policies allow people to directly see what factors of the state are involved in the choice of action, as well as the rough relationship between the action and these factors. \n(2) It has been a consensus in previous works that using symbolic expression forms of policies has better interpretability [1][2][3]. In this paper, our main motivation is to design an **efficient** symbolic policy learning method, which greatly reduces the number of interaction episodes required for learning symbolic policies and extends the application of symbolic policies to the field of meta-RL.\n(3) To further measure the interpretability of the learned symbolic policies, we present a human-study in the global rebuttal.\n\n[1] Discovering symbolic policies with deep reinforcement learning. ICML 2021  \n[2] Optimal control via reinforcement learning with symbolic policy approximation.  \n[3] Interpretable policies for reinforcement learning by genetic programming. \n\n>**Q4.** Is the main contribution of the paper the architecture ...\n\nA4. We outlined our contributions in A1. It is also possible to apply symbolic approaches to solve supervised learning problems (such as image classification and machine translation), but our symbolic policy learning method cannot be directly employed for this purpose.\n\n>**Q5.** How is q defined ? ...\n\nA5. For the definition and training methodology of Q, please refer to A2. In the experiments, the network structure for Q is a multi-layer perceptron (MLP) with a hidden layer size of [256, 256].\n\n>**Q6.** How are chosen the sequence of symbolic ...\n\nA6. Please refer to A2.\n\n>**Q7.** From my understanding, since we have m+2n inputs in the symbolic layer and only m+n outputs, the architecture has a width that reduces with layers. is it true ?\n\nA7. The architecture dose not have a width that reduces with layers. There is also a fully connected layer between the symbolic layer (line 124-125), and the output dimension of the fully connected layer is always the input dimension (m+2n) required for symbolic operations in the next layer, so the width does not gradually decrease. The width depends on the symbolic operators in the next layer.\n\n>**Q8.** What is the impact of lmin in (5) ? Do you have experiments to analyze it ?\n\nA8. To constrain the minimal complexity of symbolic policies, we introduce the minimum L0 norm denoted as $l_{min}$ (line 187). We gradually reduce $l_{min}$ from the count of the original parameters w to a specified target value using a parabolic function (lines 195-196). As training progresses, this approach leads to a gradual reduction in the complexity of symbolic policies, resulting in a compact expression. The learning curve for the complexity of symbolic policies is presented in Appendix Figure 3. We intuitively set the value for the target $l_{min}$ and do not tune it.\n\n>**Q9.** The metaRL objective given in the end of the first paragraph of page 6 is difficult to understand\n\nA9. We clarify the metaRL in the global rebuttal."}, {"review_id": "634uBzLqKY", "reviewer": "Reviewer_PhEA", "summary": "This paper proposes to apply differentiable symbolic regression for policy learning. It shows promising results in multiple RL environments (including good average performance and learned interpretable symbolic policies). ", "questions": "* Why would the symbolic methods outperform the neural ones in the single-task RL settings? \n* Are there results in more complex/practical environments, e.g., with larger input and output dimensions or longer expressions?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper shows promising results of differentiable symbolic regression in eight RL environments. The model design is intuitive and is shown to achieve good average performance, learn interpretable symbolic policies, and be less likely to overfit for OOD generalization in the meta-RL setting. The model is also more sample efficient than a previous neural symbolic regression model for policy learning. ", "weaknesses": "Nevertheless, the model is less novel as similar techniques have been explored in domains such as differentiable interpreters (e.g., Terpret [1] or DiffForth [2]) for program synthesis and Dart [3] for neural architecture search/pruning. \n\nThere are also concerns regarding the experiments:\n* Symbolic methods are reported to perform better than neural ones even for single-task RL settings, which is counter-intuitive. There should not be overfitting problems in the naive setting, as far as I understand. Why would symbolic methods outperform neural ones? \n  - I find this in appendix, lines 123-124, which seems relevant but could be unfair:\n   > For all the environments, the proposed ESPL performs 3 independent training runs and selects the single best policy.\n  - I also find that they may report averaged performance over multiple runs, which may not be necessary or standard for the single-task RL setting. \n* Overall, the descriptions regarding evaluation procedures are vague, e.g.,\n  - it's unclear whether the final policies evaluated in Table 2 are symbolic or not; and\n  - how to select policies in Table 1;\n\n\n[1] Gaunt, Alexander L., et al. \"Terpret: A probabilistic programming language for program induction.\" arXiv preprint arXiv:1608.04428 (2016).\n\n[2] Bošnjak, Matko, et al. \"Programming with a differentiable forth interpreter.\" International conference on machine learning. PMLR, 2017.\n\n[3] Liu, Hanxiao, Karen Simonyan, and Yiming Yang. \"Darts: Differentiable architecture search.\" arXiv preprint arXiv:1806.09055 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review. We are glad to discuss your concerns one by one. Any further discussion will be appreciated.\n>**Q1.** Nevertheless, the model is less novel as similar techniques have been explored in domains such as differentiable interpreters (e.g., Terpret [1] or DiffForth [2]) for program synthesis and Dart [3] for neural architecture search/pruning.\n\nA1. The proposed framework is novel and is different from previous works. Building differentiable frameworks and using gradient-based methods to obtain solutions have indeed found applications in many fields. However, compared to ESPL, these works differ significantly in terms of (1) the constructed differentiable frameworks, (2) the process of obtaining solutions through gradients, and (3) the specific problem domains they target. For example, Darts formulate a super-network based on the continuous relaxation of the neural architecture representation, while we employ a densely connected symbolic network. Darts uses the softmax function to select one proper neural network operation from a set of candidate operations for each connection. In contrast, we learn to mask out redundant connections with the proposed path selector and only a very small percentage of connections are selected. Darts aims to search for a good neural network structure for computer vision while we aim to find compact symbolic policies for reinforcement learning. In Appendix E2, we have compared our proposed method with Neural Architecture Search approaches. We will add these works into the reference. \n\n>**Q2.** Symbolic methods are reported to perform better than neural ones even for single-task RL settings, which is counter-intuitive. There should not be overfitting problems in the naive setting, as far as I understand. Why would symbolic methods outperform neural ones? I find this in appendix, lines 123-124, which seems relevant but could be unfair:'For all the environments, the proposed ESPL performs 3 independent training runs and selects the single best policy.' I also find that they may report averaged performance over multiple runs, which may not be necessary or standard for the single-task RL setting.\n\nA2. In fact, for a fair comparison, we adopted the **exact same testing procedures** as in DSP[1] which is the SOTA of symbolic policy learning and the most important baseline, including selecting from 3 independent training runs and averaging over multiple runs. It is worth noting that neural network policies are not necessarily superior to symbolic policies, and this is also evident from the experimental results in DSP[1]. One possible reason is that symbolic policies can represent complex relationships through combinations of different symbolic operations. When the symbolic expression is already sufficient to represent the policy, optimizing fewer parameters may be simpler and more effective. \n\n[1]Landajuela, Mikel, et al. \"Discovering symbolic policies with deep reinforcement learning.\" International Conference on Machine Learning. PMLR, 2021.  \n\n>**Q3.** Overall, the descriptions regarding evaluation procedures are vague, e.g., it's unclear whether the final policies evaluated in Table 2 are symbolic or not; and how to select policies in Table 1;\n\nA3. The average episode rewards of ESPL presented in Table 2 are obtained by evaluating the final obtained **symbolic policies**. Additionally, the symbolic policy given in Table 1 is the one used for evaluation. We followed the same method as in DSP to select the symbolic policy, performing independent training runs and selecting the single best policy (see Appendix lines 123-124). We will add a description in revision to make it clearer.\n\n\n>**Q4.** Are there results in more complex/practical environments, e.g., with larger input and output dimensions or longer expressions?\n\nA4. In fact, for a fair comparison, we used the exact same benchmark as in DSP (line 240-241). This benchmark includes classic control tasks such as CartPole and complex tasks like BipedalWalker, whereas earlier symbolic policy works were only evaluated on simpler tasks, such as CartPole or 1-DOF robot arm. Additionally, we included some additional experimental results (more environments and comparisons) in Appendix F."}, {"review_id": "vGaT7C9fkf", "reviewer": "Reviewer_9VjU", "summary": "#  I have reviewed this draft once before. \nThis paper proposes a meta-RL method to generate explainable symbolic policies. ESPL contains a symbolic network in the search space and a path selector to find the compact symbolic policy.\n", "questions": "1. Standard errors should be shown.\n2. Table3 is a waste of space.", "limitations": "Not discussed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think there exists some novelty in ESPL because it contains symbolic + neural structures. Also, the symbolic policies induced by ESPL seem to be effective. The number of experiments is enough. The visualizations are cool.", "weaknesses": "Post rebuttal: I upgrade my score to a borderline accept because of the reasonable rebuttal.\n\nAlthough I originally leaned to accept this paper, I think the suggestions from the previous conference are not incorporated, and the draft is not improved much, so this time, I have to vote for rejection.\n\nThe major concerns are: \n\n(1) The proposed symbolic policy looks messy and it does not improve the interpretability much. Also, there could be many optimal policies.\n\n(2) The CatPole and other approaches have guaranteed optimal policy. The theoratcal analysis is missing. \n\n(3) The authors should try other discrete environments or complicated environments with rich semantics.\n\nI feel sorry for this paper. It was very close to acceptance last time but the AC insisted that the novelties were not enough. I don't think a resubmission without major modifications can get this paper in.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your advice on further improving this paper. We would like to discuss them one by one. Any further discussion will be appreciated.\n>**Q1.**  I think the suggestions from the previous conference are not incorporated, and the draft is not improved much.\n\nThank you again for the time and effort invested in the review process. We highly value the insightful feedback from the reviewers, which has been immensely beneficial to our work. We have diligently incorporated your suggestions provided during the previous conference review process. These revisions encompass additional discussions of related works (including NAS and Programmatical RL), experiments conducted in a more complex environment (TORCS), and comparisons with some Programmatical RL methods. Due to page limitations of NeurIPS, these modifications have been placed in the appendix. The specific revisions include:\n1. Discussion of Programmatical RL and program-guided in Appendix E.1.\n2. Discussion of Neural Architecture Search in Appendix E.2.\n3. Comparison with Programmatical RL (NDPS,VIPER,PROPEL,PRL) in Appendix F.1.\n4. Experiments in TORCS in Appendix F.1.\n\n>**Q2.** The proposed symbolic policy looks messy and it does not improve the interpretability much. Also, there could be many optimal policies.\n\n1. The symbolic policies we obtained have a similar complexity compared to the previous works. In addition, the symbolic policies allow people to directly see what factors of the state are involved in the choice of action, as well as the rough relationship between the action and these factors. Besides, the superior interpretability to NN-policies has been discussed in previous works.\n2. Despite the potential existence of optimal policies in certain environments, as far as we know, obtaining an analytical expression for such policies can be quite challenging or even infeasible. Even if attainable, it requires domain-specific knowledge and intricate computations. In contrast, our approach does not necessitate prior knowledge about the environment; it autonomously learns symbolic policies through interaction with the environment.\n\n\n>**Q3.** The CatPole and other approaches have guaranteed optimal policy. The theoretical analysis is missing.\n\nFor complex nonlinear systems, we cannot obtain an analytical optimal policy. For simple systems like cartpole, since we can first know its dynamics and equilibrium points, one approach is to **approximate a linear system near the equilibrium point**, and use a linear quadratic regulator (LQR) for control. However, for other more complex environments, this becomes challenging to achieve. In contrast, our approach doesn't necessitate knowledge of the system's dynamics and can automatically learn symbolic policies. \n\nWe provide the theoretical analysis of CartPole in the global rebuttal.\n\n>**Q4.** The authors should try other discrete environments or complicated environments with rich semantics.\n\nIn this paper, we focus on tasks with continuous action space following previous works. For benchmark selection, we used the same benchmarks as in DSP to ensure a fair comparison, and we provided additional experimental results on other environments (TORCS) in the appendix.\n\n\n> **Q5.** It was very close to acceptance last time but the AC insisted that the novelties were not enough. I don't think a resubmission without major modifications can get this paper in.\n\nThis is indeed frustrating, as the author was unable to see the discussion, we do not what happened in the final discussion. However, from the Paper Decision written by the AC, we did not see any content related to novelty. In fact, apart from summarizing the reviewers' opinions, the AC believes that there is a line of transformer-based SR methods after DSP that can also improve data efficiency should be compared. However, this is factually incorrect. Symbolic regression tasks and obtaining symbolic policies in reinforcement learning tasks are not the same thing. For symbolic regression tasks, one needs to regress expressions from the data generated by a certain symbolic expression, while symbolic policy learning aims to obtain symbolic policies that maximize cumulative rewards. Specifically, transformer-based SR requires ground-truth expressions during training, which might be suitable for supervised learning but is unreasonable for reinforcement learning. Unfortunately, we did not have the opportunity to rebuttal.\n\nDue to the page limitation, most of our modifications are in the appendix. We have added discussions on related works of NAS and Programmatical RL and conducted experiments on a more complex environment (TORCS). We have also included comparisons with some Programmatical RL methods.\n\n\n>**Q6.** Standard errors should be shown.\n\nIn the table below, we give the mean and standard deviation of ESPL's episode rewards in the benchmark, but because previous works did not provide the standard deviation, we did not compare it in the table.\n\n|  | CartPole | MountainCar | Pendulum | InvDoublePend | InvPendSwingup | LunarLander | Hopper | BipedalWalker |\n|---|---|---|---|---|---|---|---|---|\n| ESPL | 1000$\\pm$0.0 | 94.02$\\pm$0.3 | -151.72$\\pm$101.31 | 9359.9$\\pm$1.54 | 890.36$\\pm$38.80 | 283.56$\\pm$20.35 | 2442.48$\\pm$25.22 | 309.43$\\pm$0.34 |\n\n>**Q7.** Table3 is a waste of space.\n\nTable 3 illustrates the number of episodes required for training. In contrast to DSP, ESPL utilizes significantly less data, showcasing its remarkable efficiency. "}]

Author rebuttal:
> **Theoretical analysis of cartpole.**

The dynamic of cartpole system can be defined as:  
$\\ddot{x}=\\frac{8fa+2m \\sin \\theta(4L\\dot{\\theta}^2-3g\\cos \\theta)}{8M-3m \\cos2\\theta+5m}$  
$\\ddot{\\theta}= \\frac{g \\sin \\theta- ( \\cos \\theta(fa+Lm \\dot{ \\theta}^2 \\sin \\theta))/(m+M)}{L( 4/3- (m \\cos^2\\theta)/(m+M))}$

Where $f$ is the coefficient of force, $a$ represents the action, $m$ is the weight of the pole, $M$ is the weight of the cart, $L$ is the half-pole length, $\\theta$ is the angle between the pole and the vertical direction, and $x$ denotes the horizontal coordinate of the cart.

Define $X=\\begin{bmatrix} x\\\\ \\dot{x}\\\\ \\theta\\\\ \\dot{\\theta}
\\end{bmatrix}$, then the derivative of $X$: $\\dot{X}=\\begin{bmatrix} \\dot{x}\\\\ \\ddot{x}\\\\ \\dot{\\theta}\\\\ \\ddot{\\theta}
\\end{bmatrix}=\\begin{bmatrix} \\dot{x}\\\\\\frac{8fa+2m\\sin\\theta(4L\\dot{\\theta}^2-3g\\cos \\theta)}{8M-3m\\cos2\\theta+5m}\\\\\\dot{\\theta}\\\\\\frac{g \\sin \\theta- ( \\cos \\theta(fa+Lm \\dot{ \\theta}^2 \\sin \\theta))/(m+M)}{L( 4/3- (m \\cos^2\\theta)/(m+M))}
\\end{bmatrix}$  
According to the Hartman-Grobman theorem, the local stability of this nonlinear system near its equilibrium point is equivalent to the linearized system near the equilibrium point. For cartpole system, the equilibrium point is $[x,\\dot{x},\\theta,\\dot{\\theta}]=[0,0,0,0]$.
If $a = 0$, the system can be linearized as:

$\\dot{X}=\\begin{bmatrix}0&1&0&0\\\\0&0&\\frac{-6gm}{8M+2m}&0\\\\0&0&0&1\\\\0&0&\\frac{g}{L(4/3-M/(m+M))}&0\\end{bmatrix}\\begin{bmatrix} x\\\\\\dot{x}\\\\\\theta\\\\\\dot{\\theta}
\\end{bmatrix}$  
Calculate its eigenvalues:
$[0, 0, 3.97114593, -3.97114593]$.
Due to the presence of positive eigenvalues, according to the Hartman-Grobman theorem, the system is unstable.  
If $a=17.17\\theta+1.2\\dot{\\theta}$ which is learned by ESPL,  
linearize the system near the equilibrium point:  
$\\dot{X}=\\begin{bmatrix}0&1&0&0\\\\0&0&\\frac{137.36f-6gm}{8M+2m}&\\frac{9.6f}{8M+2m}\\\\0&0&0&1\\\\0&0&\\frac{g-17.17f/(m+M)}{L(4/3-M/(m+M))}&\\frac{-1.2f}{L(m+M)(4/3-M/(m+M))}\\end{bmatrix}\\begin{bmatrix} x\\\\\\dot{x}\\\\\\theta\\\\\\dot{\\theta}
\\end{bmatrix}$  
Calculate its eigenvalues:
$[0+0.j, 0+0.j, -26.34+6.65014286j, -26.34-6.65014286j]$.  
Since all the real parts of the eigenvalues are non-positive, according to the Hartman-Grobman theorem, the system is stable.
Therefore, for the CartPole environment, the policies learned through ESPL can maintain the stability of the CartPole system.

> **Human study for interpretability.**

The assessment of policy interpretability actually requires some understanding of the environments and what each state variable means. We invited ten researchers to rate the interpretability of policies, with a maximum of 20 minutes for each policy. On a five-point scale, we told them that the interpretability could be judged based on whether they could see from the policy expression what the policies was based on to make decisions and the possible corresponding relationships between actions and states. A score of five indicated that the strategy was highly interpretable and could be designed by humans, while a score of zero indicated that it was completely uninterpretable, just like the neural network policies. We measured the average score of the interpretability of the policies. The interpretability score obtained by ESPL and DSP are:
|  | CartPole | MountainCar | Pendulum | InvDoublePend | InvPendSwingup | LunarLander | Hopper | BipedalWalker |
|---|---|---|---|---|---|---|---|---|
| ESPL | 5 | 5 | 4.5 | 4.1 | 4.3 | 3.9 | 2.6 | 3.1 |
| DSP | 5 | 5 | 4.5 | 5.0 | 4.2 | 4.0 | 3.1 | 3.2 |

Based on the results of the human-study, we found that when the symbolic expressions were relatively short, there was a consensus that the expression were highly interpretable. When the expression length became longer, the interpretability score decreased. It may be relatively difficult to
understand symbolic expressions in a short time, but it is generally believed that interpretability is much higher than black box, at least they can understand the action will be affected by which state, and the partial correlation.

> **Clarification of metaRL**

 We use the same metaRL objective, training, and testing settings as in previous works[1][2]. The context refers to the trajectories collected from the environment, and the context encoder derives a context variable based on this context. The context variable can be understood as a kind of identifier for the environment or task. During the training process, the policy maximizes rewards on the corresponding task based on the context variable. Therefore, we can anticipate that in a new task, as long as trajectories are collected and the context variable is obtained, metaRL can exhibit strong performance on that task. We also provide the pseudocode for the training process in Appendix Algorithm 2. Furthermore, the testing process of meta-RL is similar to your description: during testing, for a new task, the agent first undergoes an exploration phase to collect trajectories and update the context variable (lines 213-214). In subsequent phases, metaRL can achieve favorable results on the task based on the context variable. We will revise the description for clarity in the revision. Due to space constraints and the well-defined nature of the context-based metaRL process in the literature, we only provided a brief overview in the paper. We will add a detailed introduction to metaRL in the appendix.

[1] Rakelly, Kate, et al. "Efficient off-policy meta-reinforcement learning via probabilistic context variables." International conference on machine learning. PMLR, 2019.  
[2] Sarafian, Elad, Shai Keynan, and Sarit Kraus. "Recomposing the reinforcement learning building blocks with hypernetworks." International Conference on Machine Learning. PMLR, 2021


Meta-review: The reviews are unanimous. The merits of this paper is two-folded. First of all, it is important to explore symbolic methods for interpretability and safety guarantees. It is in general more accessible to have certificate over symbolic methods instead of neural ones. Secondly, the results look promising, and the proposed method is general. However, most of reviews have concerns (even after rebuttal and discussion) over two aspects of this paper, which we highly recommend this paper to continue exploring in the future. First of all, its applicability in more complex domains (aka, scalability issues). Secondly, is interpretability strongly correlated to symbolic methods. Or in other words, are symbolic methods always interpretable. As raised in a few reviews, the policy learned here sometimes still looks messy. Is it a sign of overfitting? We would like in the final version, this paper could consider having a more thorough and comprehensive discussion over those important issues.
Decision: Accept (poster)