SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- CartPole: In the environment, there is a cart that can move linearly with a pole fixed on it. The goal is to balance the pole by applying forces in the left and right direction on the cart. The max horizon length is set as 1000. We use a continuous version from an open-source implementation https://gist.github.com/ianandforth.
- MountainCar: In the environment, there is a car placed stochastically at the bottom of a sinusoidal valley. The action is the accelerating the car in either direction. The goal of the environment is to accelerate the car to reach the goal state on top of the right hill. We use MountainCarContinuous-v0 from OpenAI Gym[40].
- Pendulum: In this environment, there is a pendulum attached to a fixed point at one end. The pendulum starts in a random position and the goal is to apply torque on the free end to swing it into an upright position. We use Pendulum-v0, from OpenAI Gym.
Algorithm 1 The training process of ESPL.
Input: The number of iterations for temperature and  $L_0$  norm schedule  $t_s$ . The target temperature  $\tau_t$  and target minimum  $L_0$  norm  $l_t$ . The symbolic network  $\mathcal{SN}$  and the parameters  $\boldsymbol{w}, \boldsymbol{b}$ . The probabilities  $\pmb{p}$  of the path selector.  
1: Initialize replay buffers  $\mathcal{B}$ .  
2: for training iteration  $t = 0$  to  $T - 1$  do  
3:  $\tau = (1 - \tau_t) * (1 - \frac{\min(t, t_s)}{t_s}) + \tau_t$   
4: Generate symbolic policy  $\pi_{sym}$  with the symbolic network and the path selector.  
5: for environment step  $k = 0$  to  $K - 1$  do  
6: Collect data with  $a \sim \mathcal{N}(\pi_{sym}(s), F(s))$  and add to buffer  $\mathcal{B}$   
7: end for  
8:  $l_{min} = l_t + (1 - l_t) * \left(1 - \frac{\min(t, t_s)}{t_s}\right)^2$   
9: for steps in training step do  
10: Sample RL Batch from the buffer  $\mathcal{B}$   
11: Sample  $m$  with gumbel sigmoid and ST trick.  
12: Obtain the policy:  $\mathcal{SN}(\boldsymbol{w}_m)$   
13: Calculate loss for the Critic:  $\mathcal{L}_{critic} = \mathcal{L}_{critic}^{sac}$   
14: Calculate loss for the Actor:  $\mathcal{L}_{actor} = \mathcal{L}_{actor}^{sac} + \alpha_1 \mathcal{L}_{penalty} + \alpha_2 \mathcal{L}_{select}$   
15: end for  
16: Update the  $\theta_Q$  with  $\mathcal{L}_{critic}$   
17: Update  $w, b, p, \theta_F$  with  $\mathcal{L}_{actor}$   
18: end for
- InvertedDoublePendulum: In this environment, there is a cart that can move linearly, with a pole fixed on it and a second pole fixed on the other end of the first one. The cart can be pushed left or right, and the goal is to balance the second pole on top of the first pole by applying continuous forces on the cart. We use InvertedDoublePendulumBulletEnv-v0 from PyBullet [49].
- InvertedPendulumSwingup: This environment is a combination of CartPole and Pendulum. The goal of the environment is to swing-up the pendulum from its natural pendent position to its inverted position. We use InvertedPendulumSwingupBulletEnv-v0 from PyBullet.
- LunarLander: In this environment, there is a spacecraft with a main engine and two orientation engines. The goal of the environment is to land a spacecraft on a landing pad without crashing by controlling the engines. We use LunarLanderContinuous-v2 from OpenAI Gym.
- Hopper: In this environment, there is a two-dimensional one-legged robot that should move forward. We use HopperBulletEnv-v0 from PyBullet.
- BipedalWalker: This is a simple 4-joint walker robot environment. The goal is to control the walker to move forward. We use BipedalWalker-v2 from OpenAI Gym.
B.2 Meta-RL
We introduce the details of the environments used in the meta-RL experiments. We build all the meta-RL environments by modifying the origin environments of OpenAI Gym<sup>3</sup>. We describe these environments as follows:
- Walker2d-params: This environment is built by modifying the OpenAI gym Walker2d, in which a two-dimensional two-legged robot should move forward. For each task, the dynamics parameters of the environment are randomly initialized. The horizon length is set as 200. For the experiment, we sample 40 training tasks and 10 test tasks.
- Cheetah-vel-ood: This environment is built by modifying the OpenAI gym Half Cheetah. In the modified environment, a 2-dimensional robot with nine links should achieve a target
3We build our meta-RL environments based on the random-param-env https://github.com/dennis188/rand_param_envs.
Algorithm 2 The training process of CSP.
Input: Batch of training tasks  $\{\kappa_i\}_{i = 1,\dots m}$ . The number of iterations of temperature and target  $L_{0}$  norm schedule  $t_s$ . Target temperature  $\tau_t$ . The target  $L_{0}$  norm of the mask at the end of training  $l_t$ . The scale of the penalty loss  $\alpha_{1}$ . The scale of the loss to regularize the sum of probabilities  $\alpha_{2}$ . The scale of the loss for the KL divergence  $\beta$ .
1: Initialize replay buffers  $\mathcal{B}_i$  for each training tasks.
2: for training iteration  $t = 0$  to  $T - 1$  do
3:  $\tau = (1 - \tau_{t}) * (1 - \frac{min(t, t_{s})}{t_{s}}) + \tau_{t}$
4: for each task  $\kappa_{i}$  do
5: Initialize context  $c_{i} = \{\}$
6: for  $k = 0$  to  $K - 1$  do
7: Sample  $z\sim q_{\theta_q}(z|c_i)$
8: Generate symbolic policy  $\pi_{sym}$  with  $\Phi(z)$  and  $\Psi(z, \tau)$
9: Collect data with  $a \sim \mathcal{N}(\pi_{sym}(s), F(s, z))$  and add to buffer  $\mathcal{B}_i$ .
10: Update  $c_i = \{(s_j, a_j, s_j', r_j)\}_{j=1,\dots,N} \sim \mathcal{B}_i$
1: end for
2: end for
3: for steps in training step do
4: Sample a batch of tasks.
5: for each  $\kappa_{i}$  in the batch do
6: Sample context  $c_i$  and RL Batch  $b_i$  from the buffer  $\mathcal{B}_i$
7: Sample context variables  $z \sim q_{\theta_q}(z|c_i)$
8: Obtain parameters  $\pmb{w},\pmb{b}$  with  $\Phi (z)$
9: Obtain probabilities  $\pmb{p}$  with  $\Psi(z)$ .
20: Sample  $\mathbf{m}$  with gumbel sigmoid and ST trick.
21: Obtain the policy:  $\mathcal{SN}(\pmb{w}_m)$
22:  $l_{target} = l_t + (1 - l_t)*\left(1 - \frac{min(t,t_s)}{t_s}\right)^2$
23: Calculate loss for the critic:  $\mathcal{L}_{\text{critic}}^i = \mathcal{L}_{\text{critic}}^{sac}$
24: Calculate loss for the Actor:
$$
\mathcal {L} _ {\text {a c t o r}} ^ {i} = \mathcal {L} _ {\text {a c t o r}} ^ {s a c} + \alpha_ {1} \mathcal {L} _ {\text {p e n a l t y}} + \alpha_ {2} \mathcal {L} _ {\text {s e l e c t}}
$$
25: Calculate the KL divergence  $\mathcal{L}_{KL}^{i} = D_{KL}(q_{\theta_q}(z|c_i)|\mathcal{N}(0,1))$
26: end for
27: Update the Critic with  $\sum_{i=1}^{m} \mathcal{L}_{\text{critic}}^i$
28: Update the context encoder with  $\sum_{i=1}^{m} (\mathcal{L}_{\text{critic}}^i + \beta \mathcal{L}_{KL}^i)$
29: Update the path collector and the parameter generator with  $\sum_{i=1}^{m} \mathcal{L}_{\text{actor}}^i$
30: end for
31: end for
velocity running forward. For a training task, the target velocity is sampled uniformly from [0, 2.5]. For a test task, the target velocity is sampled uniformly from [2.5, 3.0]. The horizon length is set as 200. For the experiment, we sample 50 training tasks and 15 test tasks.
- Hopper-params: This environment is built by modifying the OpenAI gym Hopper, in which a two-dimensional one-legged robot should move forward. For each task, the dynamics parameters of the environment are randomly initialized. The horizon length is set as 200. For the experiment, we sample 40 training tasks and 10 test tasks.
- LunarLander-params: This environment is built by modifying the OpenAI gym Lunar Lander, in which a rocket should land on the landing pad. For each task, we randomly initialize the gravity. The horizon length is set as 1000. For the experiment, we sample 40 training tasks and 10 test tasks.
- InvDoublePend-params: This environment is built by modifying the OpenAI gym Inverted Double Pendulum. In this environment, there is a cart that can move linearly. A pole is fixed on it, and a second pole is fixed on the other end of the first pole. For each task, the dynamics parameters of the environment are randomly initialized. The horizon length is set as 200. For the experiment, we sample 40 training tasks and 10 test tasks.
Table 7: Measure of uncertainty for ESPL.
Environment	Measure of uncertainty
CartPole	0.4998
MountainCar	0.4998
Pendulum	0.4997
InvDoublePend	0.4995
InvPendSwingup	0.4999
LunarLander	0.4997
Hopper	0.4999
BipedalWalker	0.4998
- Cartpole-flood: This environment is built by modifying the OpenAI gym Cart Pole<sup>4</sup>. In the environment, there is a cart that can move linearly with a pole fixed on it. For each task, we randomly initialize the force magnitude and the length of the pole. For a training task, the force magnitude and the length of the pole are sampled uniformly from [7.5, 12.5] and [0.3, 0.7]. For a test task, the force magnitude and the length of the pole is sampled uniformly from [5, 7.5]  $\cup$  [12.5, 15] and [0.2, 0.3]  $\cup$  [0.7, 0.8]. The horizon length is set as 200. For the experiment, we sample 40 training tasks and 10 test tasks.
For all the environments of single-task RL and meta-RL, we wrap the action interface with the tanh function to limit the range of the action.
C.1 Single-task RL
During training, we employ two neural networks to approximate the Q function as the original SAC. The neural networks have two hidden layers with 256 hidden units. Besides, to limit the probabilities  $p$  in the range of [0, 1], we clip the probabilities after each update. After training, even if the mask is sampled from the Bernoulli distribution with probabilities  $p$ , the process of obtaining the symbolic policy is almost deterministic, because empirical observations show that almost all probabilities converge to 0 or 1 after training. We use a measure of uncertainty which is the average difference between probability  $p$  and 0.5. When the measure approaches 0.5, the process of obtaining the symbolic policy is almost deterministic. We report the measure in Table 7. For the details of the training process, we give the pseudo-code in Algorithm 1.
C.2 Meta-RL
In practice, we build up our off-policy learning framework on top of the soft actor-critic algorithm (SAC)[39] following PEARL. To construct a stochastic policy, we also employ a small neural network  $F$  to output the standard deviation just like single-task RL. The neural network has two hidden layers with 64 hidden units. Note that this neural network is only used during training. During the evaluation, we only use the produced symbolic policy to infer the action. Besides, to limit the probabilities  $p$  produced by  $\Psi(z)$  in the range of  $(0,1)$ , we employ the sigmoid function. We initialize the bias of the last layer in  $\Psi(z)$  as 3.0. As  $sigmoid(3.0) = 0.9526$ , we will have most of the paths of the symbolic network active at the beginning of training and ensure that the input of the sigmoid function is not too large to prevent the gradient from disappearing during training. For the details of the training process, we give the pseudo-code in Algorithm 2.
4We use the same continuous version as single-task CartPole.
Table 8: Hyperparameters for the ESPL.
Parameter	Value
optimizer	Adam[50]
number of samples per minibatch	256
scale of the reward	1
learning rate	3·10-4
discount	0.99
sac target smoothing coefficient	0.005
target temperature	0.2
training steps per iteration	1000
scale of the penalty loss	1
Table 9: Environment Specific Hyperparameters for ESPL.
Environment	Scale of \( \mathcal{L}_{\text{select}} \)	Target \( l_0 \) norm ratio	Schedule iterations
CartPole	0.08	0.002	400
MountainCar	0.64	0.002	200
Pendulum	0.08	0.002	300
InvDoublePend	0.08	0.005	400
InvPendSwingup	0.1	0.005	400
LunarLander	0.08	0.005	600
Hopper	0.08	0.01	2000
BipedalWalker	0.64	0.01	1400
D.1 Single-task RL
In this section, we give the main hyperparameters of ESPL for single-task RL. We show the common hyperparameters of ESPL in Table 8. We also list the environment specific hyperparameters in Table 9. For the scale of  $\mathcal{L}_{\text{select}}$ , we choose the best one from  $\{0.04, 0.08, 0.1, 0.16, 0.32, 0.64\}$ . The target  $l_0$  norm ratio is the ratio of the target  $L_0$  norm of the mask at the end of training  $l_e$  to the number of parameters of the symbolic network. We set the value according to the complexity of the task and do not tune the value. We show the learning curves of the average  $L_0$  norm ratio of the mask for the sampled tasks in Figure 8. The  $L_0$  norm ratio at the end of training is always higher than the target  $l_0$  norm ratio. Thus, the  $L_0$  norm ratio at the end of training is more affected by the scale of  $\mathcal{L}_{\text{select}}$ . The schedule iterations mean the number of iterations of temperature and target  $L_0$  norm schedule. Then we illustrate the evaluation details about symbolic policies. DSP performs 3 independent training runs with different random seeds for each dimension of the action, selecting the best symbolic policy at the end of training. Thus, for environments with n-dimension action, they perform 3n training
Table 10: Hyperparameters for the CSP.
Parameter	Value
optimizer	Adam[50]
number of samples per minibatch	256
scale of the reward	5
learning rate	3·10-4
scale of the kl divergence loss	1
discount	0.99
sac target smoothing coefficient	0.005
target temperature	0.2
training steps per iteration	2000
scale of the penalty loss	1
Table 11: Environment Specific Hyperparameters for CSP.
Environment	Meta batchsize	Scale of \( \mathcal{L}_{\text{select}} \)	Target \( l_0 \) norm ratio	Schedule iterations
Walker2d-params	10	0.25	0.01	450
Hopper-params	10	0.25	0.01	300
InvDoublePend-params	10	2.0	0.01	150
Cartpole-fl-ood	10	0.25	0.002	25
Lunarlander-g	10	0.25	0.01	60
Cheetah-vel-ood	16	2.0	0.01	300
Figure 7: Learning curves of the average  $L_{0}$  norm ratio of the mask for single-task RL.
runs and select the single best policy. The symbolic regression method repeats each experiment for a single dimension of action 10 times using different random seeds and selects the expression with the lowest mean-square error. For all the environments, the proposed ESPL performs 3 independent training runs and selects the single best policy.
D.2 Meta-RL
In this section, we give the main hyperparameters of CSP for meta-RL. We show the common hyperparameters of CSP in Table 10. We also list the environment specific hyperparameters in Table 11. The meta batchsize is the number of sampled tasks per training step. We set it according to the number of training tasks. For the scale of  $\mathcal{L}_{\text{select}}$ , we choose the best one from  $\{0.1, 0.15, 0.2, 0.25, 0.5, 1.0, 2.0\}$ . The target  $l_0$  norm ratio is the ratio of the target  $L_0$  norm of the mask at the end of training  $l_e$  to the number of parameters of the symbolic network. We set the value according to the complexity of the task and do not tune the value. We show the learning curves of the average  $L_0$  norm ratio of the mask for the sampled tasks in Figure 8. The  $L_0$  norm ratio at the end of training is always higher than the target  $l_0$  norm ratio. Thus, the  $L_0$  norm ratio at the end of training is more affected by the scale of  $\mathcal{L}_{\text{select}}$ . The schedule iterations mean the number of iterations of temperature and target  $L_0$  norm schedule. We end the schedule near the end of training but for Cartpole-fl-ood in which the policy converges quickly, we reduce the number of schedule
Table 12: Average count of all selected paths and paths selected by at least ninety percent policies.
Environment	Selected paths	Mostly selected paths
Walker2d-params	76.42	70.3
Hopper-params	21.5	20.33
InvDoublePend-params	23.2	21.0
Cartpole-fl-ood	3.06	3.0
Lunarlander-g	5.2	5.0
Cheetah-vel-ood	27.04	18.5
Figure 8: Learning curves of the average  $L_{0}$  norm ratio of the mask for the sampled tasks in Meta-RL.
iterations. We run all the experiments with five random seeds and average the results to plot the learning curve.
D.3 Platform and Device
The implementation of our ESPL and CSP is based on the pytorch[51]. Besides, we train the proposed CSP and ESPL with Nvidia V100 GPU. When evaluating the inference time, we use Intel(R) Xeon(R) Gold 5218R @ 2.10GHz CPU.
E Extended Related Work
In this section, we provide an extended review of related work.
E.1 Programmatic RL and program-guided RL
Programmatic RL represents policies as short programs to improve the interpretability of the policies. A series of works[18, 52, 19, 53, 54] obtain the programmatic policies by distilling them from neural network policies. VIPER[18] extracts neural network policies into a decision tree by imitation learning algorithm Q-DAGGER to get verifiable policies. LPP[52] also employ imitation learning but exploit the logical structure and use domain-specific language to avoid engineering each individual feature. NDPS[19] optimize the program by imitation learning from a neural network policy and employ an instance-specific syntactic model called sketches as guidance. PROPEL[53] views the unconstrained policy space as mixing neural and programmatic representations and uses mirror descent to optimize the policy. They evaluate their method in continuous control domains and show better results. LEAPS[54] learns the program embedding from given programs and uses CEM to search latent programs. MPPS[55] uses conditional variational auto-encoder and program synthesis to solve the partially observed problem and obtain the policy with MaxSAT. PRL[56] learns the programmatic policies by introducing a differentiable program derivation tree and searching the structure of the tree with gradient. Although programmatic RL methods also aim to obtain simple and interpretable policies, the ESPL proposed in this paper represents policies as symbolic or algebraic expressions which are different from programmatic RL. The different forms of symbolic expressions and programs lead to different difficulties in obtaining policies. This paper proposes a solution to the low data efficiency of the existing symbolic policy methods and extends it to meta-reinforcement learning. The scope of the application is also different. Some programmatic reinforcement learning[54, 55] interacts with the environment through DSL and is not applicable to continuous control tasks. In contrast, symbolic policy works often evaluate their methods in continuous control tasks, e.g. DSP. In addition, to further verify our method, we compared the proposed ESPL with some programmatic RL methods. Program-guided RL[57] trains the agent to
Figure 9: Ablation results of the symbolic network structure (left), the path selector(right). CSP_P means the plain structure and CSP_D means the densely connected structure. CSP_L1 means replacing the path selector with the  $L_{1}$  norm minimization.
accomplish the tasks specified by a program and use a multitask policy to accomplish subtasks of the overall task. This approach is also interpretable but dependent on a given program, rather than learning an interpretable policy.
E.2 Neural Architecture Search
Neural Architecture Search (NAS)[58, 59, 60] aims at automatically designing a neural network for a certain task and dataset and has been a promising approach to automate deep learning applications. DARTs[60] is one of the representative works of gradient-based neural architecture search [60, 61, 62]. These works formulate a super-network based on the continuous relaxation of the architecture representation, which allows efficient search of the architecture using gradient descent. Although we share the similar motivation to efficiently search by gradient descent and both have the process of selecting the appropriate substructure from a large structure, the algorithms are still quite different. These works use the softmax function to select one proper neural network operation from a set of candidate operations for each connection. In contrast, we learn to mask out redundant connections with the proposed path selector and only very small percentage of connections are selected (See Figure 7). In addition, they usually divide the computer vision dataset and alternate training structures and parameters, whereas we use a completely different training approach. Besides, these works aim to search for a good neural network structure for computer vision while we aim to find compact symbolic policies for reinforcement learning. We also involve meta-RL and develop contextual symbolic policies which can adapt to new tasks, while they usually need finetune or re-training for new tasks.
Building differentiable frameworks and using gradient-based methods to obtain solutions have indeed found applications in many fields, e.g. neural architecture search, and program induction [63, 64]. However, compared to ESPL, these works differ significantly in terms of (1) the constructed differentiable frameworks, (2) the process of obtaining solutions through gradients, and (3) the specific problem domains they target.
F More Experiment Results
F.1 Comparison with Programmatical RL
In the main body of this paper, we considered that the form of the policies will affect the final performance. Thus, except for neural network policies, we selected Regression and DSP, which also represent policies with symbolic expressions for comparison in order to ensure the fairness of comparison. We would like to compare the proposed ESPL with some programmatical RL methods to further verify our method in this section. We carry out experiments on The Open Racing Car Simulator (TORCS)[65] which provides a rich environment with input from up to 89 sensors.
We first follow the experiment settings in NDPS[19], where the controller tasks the input from 29 sensors and decides values for the acceleration, and steering. We show the lap time in seconds (lower is better) in Table 13.
Table 13: Comparison results in TORCS following experiment settings in NDPS.
CG-SPEEDWAY-1	AALBORG
ESPL	50.23	107.3
NDPS	61.56	158.87
The results demonstrate that the symbolic policies produced by ESPL achieve higher performance. NDPS learns the programmatical policies from a neural network policy and the object mismatch problem may hurt the performance. We also provide the example video in https://anonymous.4open.science/r/ESPL-8F5B.
We then follow the experiment settings in PROPEL[53], where the controller tasks the input from 29 sensors and decides values for the acceleration, steering, and brake. We carry out experiments in three maps and show the lap time in seconds (lower is better) in Table 14. In this table, VIPER[18] is a method that extracts neural network policies into a decision tree by imitation learning. The PROIR means the PID controller that is also used as the initial policy in PROPEL. PROPELTREE means PROPEL using regression trees. PROPELPROG means PROPEL using a high-level language similar to NDPS. In these three maps, the symbolic policies produced by ESPL outperform other programmatical policies and is comparable with PROPELTREE. The proposed ESPL train the symbolic policies from scratch without initial policies which are used in PROPEL. We also compare
Table 14: Comparison results in TORCS following experiment settings in PROPEL.
G-Track	E-ROAD	AALBORG
ESPL	77.31	79.67	107.94
VIPER[18]	83.60	87.53	110.57
PRIOR	312.92	322.59	244.19
NDPS[19]	108.25	126.80	163.25
PROPELPROG[53]	93.67	119.17	147.28
PROPELTREE[53]	78.33	79.39	109.83
the proposed ESPL with PRL[56] on continuous control benchmarks presented in DSP in Table 15. PRL represents the policy with program derivation tree. They use affine policy which is the linear transformation as a domain-specific programming language (DSL) for Gym environments. The proposed ESPL achieve higher performance in most environments.
Table 15: Comparison with PRL on continuous control benchmarks presented in DSP.
CartPole	MountainCar	Pendulum	InvDoublePend	InvPendSwingup	LunarLander	Hopper	BipedalWalker
ESPL	1000	94.02	-151.72	9359.9	890.36	283.56	2442.48	309.43
PRL[56]	1000	95.07	-173.56	9357.93	889.47	265.17	2226.49	270.33
F.2 Ablation Study for Meta-RL