SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiaming Guo $^{1}$  Rui Zhang $^{1}$  Shaohui Peng $^{2}$  Qi Yi $^{1,3,4}$  Xing Hu $^{1,5}$   
Ruizhi Chen $^{2}$  Zidong Du $^{1,5}$  Xishan Zhang $^{1,4}$  Ling Li $^{2,6}$  Qi Guo $^{1}$  Yunji Chen $^{1,6}$  *  
 $^{1}$  SKL of Processors, Institute of Computing Technology, CAS, Beijing, China  
 $^{2}$  Intelligent Software Research Center, Institute of Software, CAS, Beijing, China  
 $^{3}$  University of Science and Technology of China, USTC, Hefei, China  
 $^{4}$  Cambricon Technologies  
 $^{5}$  Shanghai Innovation Center for Processor Technologies, SHIC, Shanghai, China  
 $^{6}$  University of Chinese Academy of Sciences, UCAS, Beijing, China  
{guojiaming, zhangrui}@ict.ac.cn, pengshaohui@iscas.ac.cn, yiqi@mail.ustc.edu.cn  
huxing@ict.ac.cn, ruizhi@iscas.ac.cn {duzidong,zhangxishan}@ict.ac.cn  
liling@iscas.ac.cn, {guoqi,cyj}@ict.ac.cn
Deep reinforcement learning (DRL) has led to a wide range of advances in sequential decision-making tasks. However, the complexity of neural network policies makes it difficult to understand and deploy with limited computational resources. Currently, employing compact symbolic expressions as symbolic policies is a promising strategy to obtain simple and interpretable policies. Previous symbolic policy methods usually involve complex training processes and pre-trained neural network policies, which are inefficient and limit the application of symbolic policies. In this paper, we propose an efficient gradient-based learning method named Efficient Symbolic Policy Learning (ESPL) that learns the symbolic policy from scratch in an end-to-end way. We introduce a symbolic network as the search space and employ a path selector to find the compact symbolic policy. By doing so we represent the policy with a differentiable symbolic expression and train it in an off-policy manner which further improves the efficiency. In addition, in contrast with previous symbolic policies which only work in single-task RL because of complexity, we expand ESPL on meta-RL to generate symbolic policies for unseen tasks. Experimentally, we show that our approach generates symbolic policies with higher performance and greatly improves data efficiency for single-task RL. In meta-RL, we demonstrate that compared with neural network policies the proposed symbolic policy achieves higher performance and efficiency and shows the potential to be interpretable.
With the development of deep neural networks as general-purpose function approximators, deep reinforcement learning (DRL) has achieved impressive results in solving sequential decision-making tasks [1, 2]. In DRL, the policies are commonly implemented as deep neural networks which involve tremendous parameters and thousands of nested non-linear operators. Despite the excellent representation ability, the neural network (NN) is very complex, making it difficult to understand, predict the behavior and deploy with limited computational resources.
With the high academic and industrial interest in interpretable and simple RL policy, some works [3, 4, 5] propose to learn the symbolic policy which is a symbolic expression composing variables, constants, and various mathematical operators. The symbolic policy has a succinct form and low complexity which is considered to be more interpretable and easily deployable in real-world settings. [3] and [4] approximate a symbolic policy with genetic programming but are limited to simple tasks with provided or learned world model and suffered from performance decrease compared with NN policies. DSP [5], the state-of-the-art symbolic policy learning method, removes some limitations by employing a recurrent neural network(RNN) as an agent to generate the symbolic policy and trains the RNN with reinforcement learning. However, this method has low data efficiency and requires hundreds of times of environment interactions compared with traditional reinforcement learning algorithms. For environments with multidimensional action spaces, they need a pre-trained NN policy as the anchor model, which brings additional complexity and may limit the final performance. In addition, the complexity of these algorithms makes it difficult to apply them to complex reinforcement learning tasks, e.g. meta-reinforcement learning.
In this paper, we propose an efficient gradient-based learning method called ESPL (Efficient Symbolic Policy Learning) for learning the symbolic policy from scratch in an end-to-end differentiable way. To express the policy in a symbolic form, the proposed ESPL consists of a symbolic network and a path selector. The symbolic network can be considered as a full set of candidate symbolic policies. In the symbolic network, the activation functions are composed of various symbolic operators and the parameters can be regarded as the constants in the symbolic expression. The path selector chooses the proper compact symbolic form from the symbolic network by adaptively masking out irrelevant connections. We design all these modules to be differentiable and represent the policy with a differentiable symbolic expression. Then we can efficiently train the symbolic policy in an off-policy manner and the symbolic policy is directly updated via gradient descent without an additional agent. Experimentally, on several benchmark control tasks, our algorithm is able to produce well-performing symbolic policy while requiring thousands of times fewer environmental interactions than DSP.
Meta-reinforcement learning (meta-RL) is one of the most important techniques for RL applications, which improves the generalization ability on unseen tasks by learning the shared internal structure across several tasks. We raise the question: is it possible to exploit the benefit of the symbolic policy and meta-RL to generate symbolic policies for unseen tasks? While previous symbolic policy methods are too complex to be combined with meta-RL, we combine the proposed ESPL with context-based meta-RL and develop the contextual symbolic policy (CSP). Context-based meta-RL [6, 7, 8, 9] is the most promising meta-RL method which forces the policy to be conditional on context variables that are formed by aggregating experiences. In the CSP framework, the path selector decides the symbolic form based on the context variables. We also involve a parameter generator to generate the constants of the symbolic policy based on the context variables. We build the training process on top of the context-based meta-RL method PEARL [7]. The proposed CSP can generate symbolic policies for unseen tasks given a few trajectories. We find that compared with neural network policies, contextual policies produced by CSP achieve higher generalization performance, efficiency, and show the potential to be interpretable.
The contributions of this paper are three-fold. First, we introduce a novel gradient-based symbolic policy learning algorithm named ESPL that learns the symbolic policy efficiently from scratch. Next, with ESPL we develop the contextual symbolic policy for meta-RL, which can produce symbolic policies for unseen tasks. Finally, we summarize our empirical results which demonstrate the gain of ESPL both in single-task RL and meta-RL. Importantly, we find empirically that contextual symbolic policy improves the generalization performance in PEARL.
2.1 Symbolic Policy
The emergence of symbolic policy is partly credited to the development of symbolic regression which is applicable in wide fields, e.g. discovering physics lows [10] and automated CPU design [11]. Symbolic regression aims to find symbolic expressions to best fit the dataset from an unknown fixed function. A series of methods [12, 13, 14] employ genetic programming (GP) to evolve the symbolic expressions. With the development of neural network and gradient descent, some methods [15, 16, 17] involve deep learning for symbolic regression. Some works employ symbolic regression
Figure 1: Example network structures for the symbolic network. Left: the plain structure. Middle: a symbolic work with dense connections. Right: a symbolic network with dense connections and arranged operators.
methods to obtain symbolic policies for efficiency and interpretability. [3] and [4] aim to approximate a symbolic policy with genetic programming but require a given dynamics equations or a learned world model. DSP [5], following the symbolic regression method DSR [16], employs a recurrent neural network to generate the symbolic policy. They use the average returns of the symbolic policies as the reward signal and train the neural network with risk-seeking policy gradients. However, for environments with multidimensional action spaces, they need a pre-trained neural network policy as the anchor model. Besides, in this framework, a single reward for reinforcement learning involves many environmental interactions, which is inefficient and makes it hard to combine the symbolic policy with meta-RL. Recently, some works [18, 19] attempt to distill an interpretable policy from a pre-trained neural network policy but have a problem of objective mismatch [5]. Different from the above-mentioned methods, we propose an efficient gradient-based framework to obtain the symbolic policy without any pre-trained model.
2.2 Meta-Reinforcement Learning
Meta-RL extends the notion of meta-learning [20, 21, 22] to the context of reinforcement learning. Some works [23, 24, 25] aim to meta-learn the update rule for reinforcement learning. We here consider another research line of works that meta-train a policy that can be adapted efficiently to a new task. Several works [26, 27, 28] learn an initialization and adapt the parameters with policy gradient methods. However, these methods are inefficient because of the on-policy learning process and the gradient-based updating during adaptation. Recently, context-based meta-RL [6, 7, 29] achieve higher efficiency and performance. For example, PEARL [7] proposes an off-policy meta-RL method that infers probabilistic context variables with experiences from new environments. Hyper [29] proposes a hypernetwork where the primary network determines the weights of a conditional network and achieves higher performance. Most of the subsequent context-based meta-RL methods [30, 31, 32] attempt to achieve higher performance by improving the context encoder or the exploration strategy. In this paper, we combine the symbolic policy with meta-RL to form the CSP and consequently improve the efficiency, interpretability and performance of meta-RL. As far as we know, we are the first to learn the symbolic policy for meta-RL.
Our method is also related to some neural architecture search methods and programmatic RL methods. We provide an extended literature review in Appendix E.
3 Gradient-based Symbolic Policy Learning
This section introduces the structure of the proposed ESPL, an end-to-end differentiable system. The proposed ESPL consists of two main components: 1) the Symbolic Network, which expresses the policy in a symbolic form, and 2) the Path Selector, which selects paths from the symbolic network to form compact symbolic expressions.
3.1 Densely Connected Symbolic Network
To construct a symbolic policy in an end-to-end differentiable form, we propose the densely connected symbolic network  $\mathcal{SN}$  as the search space for symbolic policies. Inspired by previous differentiable symbolic regression methods [17, 33], we employ a neural network with specifically designed units, which is named symbolic network. We now introduce the basic symbolic network named plain structure which is illustrated in Figure 1. The symbolic network is a feed-forward network with  $L$  layers. Different from traditional neural networks, the activation functions of the symbolic network are replaced by symbolic operators, e.g. trigonometric and exponential functions. For the  $l_{th}$  layer of the symbolic network, we denote the input as  $x_{l-1}$  and the parameters as weights  $w_l$  and biases  $b_l$ . These parameters serve as the constants in a symbolic expression. We assume that the  $l_{th}$  layer contains  $m$  unary functions  $\{g_1^1, \dots, g_m^1\}$  and  $n$  binary functions  $\{g_1^2, \dots, g_n^2\}$ . Firstly, the input of the  $l_{th}$  layer will be linearly transformed by a fully-connected layer:
$$
y _ {l} = F _ {l} \left(x _ {l - 1}\right) = w _ {l} x _ {l - 1} + b _ {l}. \tag {1}
$$
The fully-connected layer realizes the addition and subtraction in symbolic expressions and produces  $m + 2n$  outputs. Then the outputs will go through the symbolic operators and be concatenated to form the layer output:
$$
G _ {l} \left(y _ {l}\right) = \left[ g _ {1} ^ {1} \left(y _ {l} ^ {1}\right), \dots , g _ {m} ^ {1} \left(y _ {l} ^ {m}\right), g _ {1} ^ {2} \left(y _ {l} ^ {m + 1}, y _ {l} ^ {m + 2}\right), \dots , g _ {n} ^ {2} \left(y _ {l} ^ {m + 2 n - 1}, y _ {l} ^ {m + 2 n}\right) \right] \tag {2}
$$
Then the  $l_{th}$  layer of the symbolic network can be formulated as  $\mathcal{SN}_l: x_l = G_l(F_l(x_{l-1}))$ . Following the last layer, a fully-connected layer will produce a single output. For multiple action dimensions, we construct a symbolic network for each dimension of action.
Symbolic operator. The symbolic operators are selected from a library, e.g.  $\{sin, cos, exp, log, \times, \div\}$  for continuous control tasks. For the plain structure, we include an identical operator which retains the output of the previous layer to the next layer in the library. To find the symbolic policy via gradient descent, it is critical to ensure the numerical stability of the system. However, this is not natural in a symbolic network. For example, the division operator and the logarithmic operator will create a pole when the input goes to zero and the exponential function may produce a large output. Thus, we regularize the operators and employ a penalty term to keep the input from the "forbidden" area. For example, the logarithmic operator  $y = \log(x)$  returns  $\log(x)$  for  $x > \text{bound}_{\text{log}}$  and  $\log(\text{bound}_{\text{log}})$  otherwise and the penalty term is defined as  $\mathcal{L}_{\text{log}} = \max(\text{bound}_{\text{log}} - x, 0)$ . The division operator  $c = a / b$  returns  $a / b$  for  $b > \text{bound}_{\text{div}}$  and 0 otherwise. The penalty term is defined as  $\mathcal{L}_{\text{div}} = \max(\text{bound}_{\text{div}} - b, 0)$ . The details of all regularized operators can be found in the Appendix. To ensure the numerical stability, we involve a penalty loss function  $\mathcal{L}_{\text{penalty}}$  which is the sum of the penalty terms of all  $N$  regularized operators in symbolic networks:
$$
\mathcal {L} _ {\text {p e n a l t y}} = \sum_ {i = 1} ^ {i = N} \mathcal {L} _ {g _ {i}} \left(x _ {i}\right). \tag {3}
$$
Dense connectivity. We introduce dense connections [34] in the symbolic network, where inputs of each layer are connected to all subsequent layers. Consequently, the  $l_{th}$  layer of the symbolic network will receive the environment state  $s$  and the output of all preceding layers  $x_{1},\dots ,x_{l - 1}: x_{l} = G_{l}(F_{l}([s,x_{1},\dots ,x_{l - 1}]))$ . The dense connections improve the information flow between layers and benefit the training procedure. Besides, with these dense skip connections across layers, the combination of symbolic operators becomes more flexible, making the symbolic network more likely to contain good symbolic policies. In addition, we can flexibly arrange the position of operators. For example, if we only arrange the sin operator in the last layer but the oracle expression contains terms like  $\sin (s_0)$ , the input of the sin operator can still be from the original state because of the dense connections. We give an example of arranged operators in Figure 1 which we use for all tasks in the experiments. In this symbolic network, we heuristically involve more multiplication and division operators at shallow layers to provide more choice of input processed by simple operators for complex operations such as sines and cosines.
3.2 The Path Selector
The symbolic network serves as a full set of the search space of symbolic expressions. To select the proper paths from the symbolic network to produce a compact symbolic policy, we reduce the
number of paths involved in the final symbolic policy then proper paths remain and redundant paths are removed. This can be naturally realized by minimizing the  $L_{0}$  norm of the symbolic network weights. As the  $L_{0}$  norm is not differentiable, some methods [17, 33] employ  $L_{1}$  norm instead of  $L_{0}$  norm. However,  $L_{1}$  will penalize the magnitude of the weights and result in performance degradation. Inspired by the probability-based sparsification method [35, 36, 37], we propose a probabilistic path selector which selects paths from the network by multiplying a binary mask on the weights of the symbolic network  $\pmb{w}$ . The binary mask  $m_{i}$  is sampled from the Bernoulli distribution:  $m_{i} \sim \text{Bern}(p_{i})$ , where  $p_{i} \in [0,1]$  serves as the probability. Then the final weights of the symbolic network are  $\pmb{w}_{m} = \pmb{w} \otimes \pmb{m}$ , where  $\otimes$  is the element-wise multiply operation. Consequently, to get a compact symbolic expression, we only need to minimize the expectation of the  $L_{0}$  norm of the binary mask  $\mathbb{E}_{m \sim \text{Bern}(\pmb{m}|\pmb{p})} \| \pmb{m} \|_{0} = \sum p_{i}$ , without penalizing the magnitude of the weights. During the process of collecting data or testing, we can directly sample the binary mask from the Bernoulli distribution. Then we can obtain the symbolic policy  $\pi_{sym}$  by removing paths with zero weight and simplifying the symbolic expression.
However, the sampling process does not have a well-defined gradient. Thus, for the training process we build up our sampling function with the gumbel-softmax trick [38]. As the mask  $m$  is binary categorical variables, we replace the softmax with sigmoid and named the sampling function as gumbel sigmoid. The gumbel sigmoid function can be formulated as:
$$
\boldsymbol {m} _ {\boldsymbol {g s}} = \operatorname {s i g m o i d} \left(\frac {\log \left(\frac {\boldsymbol {p}}{1 - \boldsymbol {p}}\right) + \boldsymbol {g} _ {\mathbf {1}} - \boldsymbol {g} _ {\mathbf {0}}}{\tau}\right), \tag {4}
$$
where  $g_{1}$  and  $g_{0}$  are i.i.d samples drawn from Gumbel(0,1).  $\tau$  is the temperature annealing parameter. Note that  $m_{gs}$  is still not a binary mask. To obtain a binary mask but maintain the gradient, we employ the Straight-Through (ST) trick:  $m = \mathbb{1}_{\geq 0.5}(m_{gs}) + m_{gs} - \overline{m_{gs}}$ , where  $\mathbb{1}_{\geq 0.5}(x) \in \{0,1\}^n$  is the indicator function and the overline means stopping the gradient. During training, we do not remove paths with zero weight and directly use symbolic network  $\mathcal{SN}(w_m)$  as the policy.
We also involve a loss function  $\mathcal{L}_{\text{select}}$  to regularize the sum of probabilities  $p$  which is the expectation of the  $L_0$  norm of the binary mask  $m$ . To limit the minimum complexity of symbolic policies, we involve the minimum  $L_0$  norm defined as  $l_{min}$ . Then the loss function can be defined as:
$$
\mathcal {L} _ {\text {s e l e c t}} = \max  \left(\sum p _ {i} - l _ {\min }, 0\right). \tag {5}
$$
In practice, we build our off-policy learning framework on top of the soft actor-critic algorithm (SAC) [39]. We employ the neural network  $Q(s,a)$  parameterized by  $\theta_{Q}$  as the critic (state-action-value function). To construct a stochastic policy, we also employ a small neural network  $F(s)$  parameterized by  $\theta_{F}$  to output the standard deviation. Note that  $Q(s,a)$  and  $F(s)$  are only used during training. We optimize the weights  $w$ , the biases  $b$  of the symbolic network, the probabilities  $p$  in the path selector, and  $\theta_{F}$  with the combination of actor loss from SAC,  $\mathcal{L}_{\text{penalty}}$  and  $\mathcal{L}_{\text{select}}$ . We update  $\theta_{Q}$  with the critic loss from SAC. During training, we decrease the temperature parameter  $\tau$  of gumbel sigmoid linearly and decrease the  $l_{min}$  from the count of the original parameters  $w$  to a target value with a parabolic function. We summarize the training procedure and give the pseudo-code in Appendix C.
4 Contextual Symbolic Policy for Meta-RL
4.1 Background
In the field of meta-reinforcement learning (meta-RL), we consider a distribution of tasks  $p(\kappa)$  with each task  $\kappa \sim p(\kappa)$  modeled as a Markov Decision Process(MDP). In common meta-RL settings, tasks share similar structures but differ in the transition and/or reward function. Thus, we can describe a task  $\kappa$  with the 6-tuple  $(\mathcal{S},\mathcal{A},\mathcal{P}_{\kappa},\rho_{0},r_{\kappa},\gamma)$ . In this setting,  $\mathcal{S} \subseteq \mathbb{R}^n$  is a set of n-dimensional states,  $\mathcal{A} \subseteq \mathbb{R}^{m}$  is a set of m-dimensional actions,  $\mathcal{P}_{\kappa}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to [0,1]$  is the state transition probability distribution,  $\rho_0: \mathcal{S} \to [0,1]$  is the distribution over initial states,  $r_{\kappa}: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$  is the reward function, and  $\gamma \in (0,1)$  is the per timestep discount factor. Following the setting of prior works [7, 8], we assume there are  $M$  meta-training tasks  $\{\kappa_m\}_{m = 1,\dots ,M}$  sampled from the training
Table 1: Symbolic policies produced by ESPL.
Environment	ESPL
CartPole	a1=17.17s3+1.2s4
MountainCar	a1=8.06sin(9.73s2-0.18)+1.26
Pendulum	a1=- (4.27s1+0.62)(1.9s2+0.42s3)
InvDoublePend	a1=12.39s5-4.48sin(0.35s2+4.51s5+1.23s6+7.97s8+1.23s9+0.08)+0.34
InvPendSwingup	a1=4.33sin(0.17*s1+0.14s2+0.49s3+1.76s4+0.33s4-0.29)-0.65
LunarLander	a1=(0.14-2.57s4)(0.48-0.68log(0.5s2))-1.44a2=-5.72s3+4.42sin(2.54s5+0.03)-0.4-6.5s6-2.13cos(0.78sin(4.15s1-0.05)+1.98)-0.984.71*s7+0.77
Hopper	a1=-0.32s12-1.46s8-0.83s10-0.11sin(0.26s11-5s13-2.57s6+0.38)-0.92a2=-0.52s12-3.63s4-4.58s8+0.68exp(-7.31s11-2.5s13)+0.58+1.33-0.44s13a3=0.83+1.12s1-0.47-0.1exp((10.05s1-1.76s6+1.65)(0.22s13-1.88s14+1.32))(5.59*s1-0.08)/0.23+0.21exp((10.05s1-1.76s6+1.48)(0.22s13-1.88s14+1.32))
BipedalWalker	a1=1.45-2.94cos(-0.73s5+(0.06-1.06s3)(-1.33s12-0.28s6+0.41)+1.32)a2=7.53exp(0.4s1-0.13s6-0.52sin(1.5s7-0.24))-11.1a3=-4.56s9+(0.2-3.01s21)(-1.8s1-0.03s7-0.96)-0.54a4=-0.28+3.32s12+5.64s3+0.29s22-2.463.26s23-1.45
Table 2: Performance comparison of symbolic policies and neural policies for seven different DRL algorithms.
Environment	DDPG	TRPO	A2C	PPO	ACKTR	SAC	TD3	Regression	DSP	ESPL
Cartpole	1000	1000	1000	1000	1000	1000	1000	211.82	1000	1000
Mountaincat	95.36	93.6	93.97	93.76	93.79	94.68	93.87	95.16	99.11	94.02
Pendulum	-155.6	-145.49	-157.59	-160.14	-201.57	-154.82	-155.06	-1206.9	-155.4	-151.72
InvDoublePend	9347.1	9188.43	9359.81	9356.59	9359.06	9359.92	9359.25	637.2	9149.9	9359.9
InvPendSwingup	891.48	892.9	254.71	890.1	890.11	891.32	892.25	-19.21	891.9	890.36
LunarLander	266.05	265.26	238.51	269.65	271.53	276.92	272.13	56.08	261.36	283.56
Hopper	1678.84	2593.56	2104.98	2586.56	2583.88	2613.16	2743.9	47.35	2122.4	2442.48
BipedalWalker	209.42	312.14	291.79	287.43	309.57	308.31	314.24	-110.77	311.78	309.43
Worst Rank	9	10	9	9	9	6	7	10	9	6
Average Rank	5.5	4.125	6.25	6.125	5.375	3	2.875	9.125	4.625	3.5