SlowGuess/NeurIPS_2023_OCR huggingface dataset:
tasks distribution  $p_{train}(\kappa)$ . For meta-testing, the tasks are sampled from the test tasks distribution  $p_{test}(\kappa)$ . The two distributions are usually the same in most settings but can be different in out-of-distribution(OOD) settings. We denote context  $c_{T} = \{(s_{1},a_{1},s_{1}^{\prime},r_{1}),\dots ,(s_{T},a_{T},s_{T}^{\prime},r_{T})\}$  as the collected experiences. For context-based meta-RL, the agent encodes the context into a latent context variable  $z$  with a context encoder  $q(z|c)$  and the policy  $\pi$  is conditioned on the current state and the context variable  $z$ . During adaptation, the agent first collects experiences for a few episodes and then updates the context variables. Then the contextual policy is able to adapt to new tasks according to the context variables. The meta-RL objective can be formulated as  $max\mathbb{E}_{\kappa \sim p(\kappa)}[\mathbb{E}_{c_T\sim \pi}[R(\kappa ,\pi ,q(z|c_T))]]$ , where  $R(\kappa ,\pi ,q(z|c_T))$  denotes the expected episode return.
4.2 Incorporating the Context Variables
To quickly adapt to new tasks, we need to incorporate the context variables  $z \sim q(z|c_{\kappa})$  to the symbolic network and produce different symbolic policies for different tasks  $\kappa$  sampled from the task distribution  $p(\kappa)$ . To condition the parameters of the symbolic expression on the context variable, we propose a parameter generator:  $\boldsymbol{w}, \boldsymbol{b} = \Phi(z)$  which is a neural network to produce the parameters of symbolic networks for all action dimensions based on the context variables. We also involve a neural network to generate the probabilities of the path selector:  $\boldsymbol{p} = \Psi(z)$ . Then the contextual symbolic network can generate different symbolic expression forms according to the context variables.
Figure 2: Comparison for different kinds of contextual policies on meta-RL tasks. We show the mean and standard deviation of returns on test tasks averaged over five runs.
4.3 Training Schedule
We train the CSP in an off-policy manner. For meta-training epoch  $t$ , the agent first collects experiences of different training tasks into the corresponding buffer  $\mathcal{B}_{\kappa_i}$  for several iterations. At the beginning of each data collection iteration, we sample context  $c_{T}$  from buffer  $\mathcal{B}_{\kappa_i}$  and sample context variables  $z \sim q(z|c_{T})$  as PEARL [7] does. The difference is that we also sample the symbolic policy with  $\Phi (z)$  and  $\Psi (z)$  and use the sampled policy for the following steps of the iteration. For training, we sample RL batch and context from the buffer and optimize the context encoder  $q(z|c_{T})$  to recover the state-action value function. For each training step, we sample a new symbolic policy. We employ the soft actor-critic to optimize the state-action value function. For the parameter generator and the path selector, we employ  $\mathcal{L}_{select}$  and  $\mathcal{L}_{penalty}$  in addition to the SAC loss. During training, we decrease the temperature parameter  $\tau$  and  $l_{min}$  just like single-task RL. More details and the pseudo-code can be found in Appendix C.
Environment. For single-task RL, we evaluated our method on benchmark control tasks which are presented in DSP: (1) CartPole; (2) MountainCar; (3) Pendulum; (4) InvertedDoublePendulum; (5) InvertedPendulumSwingup; (6) LunarLander; (7) Hopper; (8) BipedalWalker. For meta-RL, we evaluate the CSP on several continuous control environments which are modified from the environments of OpenAI Gym [40] to be meta-RL tasks similar to [7, 9, 30]. These environments require the agent to adapt across dynamics (random system parameters for Hopper-params, Walker2d-params, Lunarlander-params, InvDoublePend-params, different force magnitude and pole length for Cartpole-fl-ood) or reward functions (target velocity for Cheetah-vel-ood).
Methods. In the single-task RL experiments, for the neural network policies, we compare our method with seven state-of-the-art DRL algorithms: DDPG, TRPO, A2C, PPO, ACKTR, SAC, and TD3 [41, 42, 43, 44, 45, 39, 46]. The results are obtained with the tuned pretrained policies from an open-source repository Zoo [47]. For symbolic policies, we include the Regression method and DSP. The Regression policies are produced with two steps: 1) generate a dataset of observation action trajectories from the best pre-trained policy from Zoo; 2) perform deep symbolic regression [16] on the dataset and select the expression with the lowest error for each action. DSP first trains a recurrent neural network with reinforcement learning to produce symbolic policies and then optimizes the constant with several methods such as Bayesian Optimization [48].
Table 3: The number of environment episodes used for learning symbolic policies.
Environment	Regression	DSP	ESPL
CartPole	1000	2M	500
MountainCar	1000	2M	500
Pendulum	1000	2M	500
InvDoublePend	1000	2M	500
InvPendSwingup	1000	2M	500
LunarLander	1000	0.4M	1000
Hopper	1000	0.4M	3000
BipedalWalker	1000	0.4M	2000
Evaluation. Following DSP, we evaluate the proposed ESPL by averaging the episodic rewards across 1,000 episodes with different environment seeds. The evaluation for all the baselines is also in accordance with this protocol. For the DSP and regression method, we use the results from the DSP paper. DSP performs 3n independent training runs for environments with n-dimension action and select the best symbolic policy. For a fair comparison, we perform three independent runs and select the best policy for ESPL and DRL methods. For meta-RL, we run all environments based on the off-policy meta-learning framework proposed by PEARL [7] and use the same evaluation settings. We compare CSP with PEARL which concatenates the observation and context variables as the input of policy and Hyper [29] which generate the parameters of policy with a ResNet model based on the context variables. Note that the original Hyper also modifies the critic, but we build all the critics with the same network structure for consistency. More details of the experimental settings can be found in Appendix D.
5.2 Comparisons for Single-task RL
Performance. In Table 2, we report the average episode rewards of different algorithms. Among symbolic policies, the proposed ESPL achieves higher or comparable performance compared with DSP while the regression method performs poorly in most environments. To directly compare ESPL with other algorithms across different environments, we rank the algorithms and calculate the average rank and worst-case rank. We find that TD3 and SAC outperform other algorithms. For symbolic policy methods, the proposed ESPL achieves superior performance compared to DSP. ESPL is also comparable with the better-performing algorithms of DRL.
Data efficiency. In Table 3, we report the number of environment episodes required for learning the symbolic policy in different algorithms. The proposed ESPL requires fewer environmental interactions to learn the symbolic policies in most environments. Compared with DSP, the proposed ESPL uses thousands of times fewer environment episodes.2 Although the regression method requires a similar number of environment episodes as ESPL, it fails to find meaningful policies in most environments according to Table 2. Besides, the proposed ESPL is trained from scratch while the regression method and DSP need pretrained neural network policies. In conclusion, the proposed ESPL greatly improves the data efficiency of symbolic policy learning.
5.3 Comparisons for Meta-RL
Performance. For meta-RL tasks, we report the learning curves of undiscounted returns on the test tasks in Figure 2. We find that CSP achieves better or comparable performance in all the environments compared with previous methods. In Hopper-params, Walker2d-params and InvDoublePend-params, CSP outperforms PEARL and Hyper during the whole training process. In Lunarlander-params, CSP achieves better final results. In Cartpole-fl-ood, CSP adapts to the optimal more quickly. In the out-of-distribution task Cheetah-vel-ood, we find the performance of PEARL and Hyper decrease during training because of over-fitting. But our CSP is less affected. Thus, expressing the policy in the symbolic form helps improve the generalization performance.
Deploying efficiency. We also evaluate the deploying efficiency of contextual policies. We first calculate the flops of each kind of policy per inference step. Then we consider an application scenario
2For DSP, we only record the environment episodes required for reinforcement learning as the exact number of iterations for constant optimization is not provided in DSP.
Table 4: FLOPs (k)/Inference time (ms) of different contextual policies.
Environment	CSP	PEARL	Hyper
Walker2d-params	3.11/20.9	189.3/27.0	5.64/22.6
Hopper-params	0.51/4.13	186.9/26.6	4.1/17.2
InvDoublePend-params	0.039/0.37	186.0/25.1	3.59/12.3
Cartpole-fl-ood	0.004/0.042	183.9/23.9	1.79/9.08
Lunarlander-g	0.015/0.14	185.4/23.4	3.08/12.3
Cheetah-vel-ood	0.53/4.9	190.2/28.4	7.18/24.2
that the algorithm control five thousand simulated robots with the Intel(R) Xeon(R) Gold 5218R @ 2.10GHz CPU and record the elapsed time per inference step. We report the results in Table 4. Compared to PEARL, CSP reduces the FLOPs by 60-45000x and reduces the inference time by up to 600x. Compared to Hyper, CSP reduces the flops by 2-450x and reduces the inference time by up to 200x. Thus, compared with pure NN policies, the CSP has a significant advantage in computational efficiency.
5.4 Analysis of symbolic policies
Interpretability. For single-RL tasks, we report the symbolic policies found by ESPL for each environment in Table 1. The policies in the symbolic form are simple and we can directly glean insight from the policies by inspection, or in other words, interpretability. For example, the goal of LunarLander is to land a spacecraft on a landing pad without crashing. The action  $a_1$  controls the main engine and  $a_2$  controls the orientation engines. In our symbolic policy,  $a_1$  is a function of  $s_1$  (the height) and  $s_4$  (the vertical velocity). The main engine turns on to counteract downward motion when the height is low. Action  $a_2$  is a combination of a term about  $s_3$  (the horizontal velocity), a term about  $s_5$  (the angle), and a highly nonlinear term about  $s_6$  (the angular velocity),  $s_1$  (the horizontal position) and  $s_7$  (whether the leg has landed). Thus, the policy adjusts the orientation engines based on the incline and horizontal motion to move the spacecraft to the center of the landing pad. In meta-RL, the CSP also shows the potential to be interpretable. We take the Cartpole-fl-ood environment as an example and illustrate the Cartpole system in Figure 3. The form of the symbolic policies produced by CSP is action =  $c_1 * \theta + c_2 * \dot{\theta} + b$ , where  $c_1$  and  $c_2$  are the positive coefficients and  $b$  is a small constant which can be ignored. Then the policy can be interpreted as pushing the cart in the direction that the pole is deflected or will be deflected. To analyze the difference between policies for different tasks, we uniformly set the force magnitude and the length of the pole. Then we generate the symbolic policy with CSP and record the coefficients. As Figure 4 shows,  $c_1$  and  $c_2$  tend to increase when the force magnitude decreases and the length increases, which is in accord with our intuition. We also provide the human study results of the interpretability in Appendix F.3.
Complexity. We compare the length of the symbolic policies and define  $\text{length} = \frac{\sum_{i=1}^{n} N_o^i + N_c^i + N_v^i}{n}$ , where  $n$  is the dimension of the action,  $i$  is the index of the action dimension,  $N_o^i$  is the number of operators,  $N_c^i$  is the number of constant terms,  $N_v^i$  is the number of variable terms. We give a comparison of the length of the symbolic policies in Table 5.
Table 5: Comparison of the length of the symbolic policies.
Average	CartPole	MountainCar	Pendulum	InvDoublePend	InvPendSwingup	LunarLander	Hopper	BipedalWalker
ESPL	12.91	3	6	7	15	13	16.5	24.6	17
DSP	8.25	3	4	8	1	19	6.5	12	12.5
In the benchmark environments used in the literature, in some environments ESPL produces longer symbolic policies than DSP, in others ESPL produces similar or shorter symbolic policies than DSP. In general, symbolic policies produced by ESPL are only slightly longer than the symbolic policies produced by DSP.
5.5 Ablation
Finally, we carry out experiments by ablating the features of the proposed ESPL. We change the structure of the symbolic network and replaced the path selector with the  $L_{1}$  norm minimization. We
Figure 3: The Cartpole system to be Figure 4: The coefficients of symbolic policies for Cartpole environment controlled.
Table 6: Ablation results for single-task RL.  $\mathrm{ESPL_p}$  and  $\mathrm{ESPL_d}$  means replacing the symbolic network with a plain structure and a densely connected structure respectively.  $\mathrm{ESPL}_{\mathrm{l}_1}$  means replacing the path selector with the  $L_{1}$  norm minimization.
Environment	ESPL	\( ESPL_p \)	\( ESPL_d \)	\( ESPL_{1} \)
CartPole	1000	1000	1000	1000
MountainCar	94.02	93.69	93.83	94.17
Pendulum	-151.72	-183.16	-144.08	-163.54
InvDoublePend	9359.9	9357.6	9197.44	8771.18
InvPendSwingup	890.36	844.84	890.01	865.38
LunarLander	283.56	263.95	277.36	271.69
Hopper	2442.48	2003.24	2316.54	1546.35
BipedalWalker	309.43	-11.63	298.50	6.81
report the average episode rewards for single-task RL in Table 6. As the experiment results show, without the path selector or the dense connections, the performance degrades, especially in Hopper and BipedalWalker. With the path selector or the dense connections,  $\mathrm{ESPL_d}$  is able to perform well for all the environments while we observe that the arranged symbolic operators can further improve the performance. We also provide the ablation study results for meta-RL in Appendix F.2.
In this paper, we introduce ESPL, an efficient gradient-based symbolic policy learning method. The proposed ESPL is able to learn the symbolic policy from scratch in an end-to-end way. The experiment results on eight continuous control tasks demonstrate that the approach achieves comparable or higher performance than both NN-based policies and previous symbolic policies while greatly improving the data efficiency compared with the previous symbolic policy method. We also combine our method with meta-RL to generate symbolic policies for unseen tasks. Empirically, compared with neural network policies, the proposed symbolic policy achieves higher performance and efficiency and shows the potential to be interpretable. We hope ESPL can inspire future works of symbolic policy for reinforcement learning or meta-reinforcement learning. Besides, as the symbolic policy is a white box and more dependable, the proposed ESPL may promote applications of reinforcement learning in industrial control and automatic chip design.
Limitations and future work. In this paper, we focus on continuous control tasks with low-dimensional state space. The proposed ESPL and CSP can not directly generate a symbolic policy for tasks with high-dimensional observation like images. A possible method is to employ a neural network to extract the environmental variables and generate symbolic policy based on these environmental variables. We leave this in the future work. Symbolic policies generally have good interpretability. However, when the task is too complex, the symbolic policy is also more complex, making the interpretability decrease. Solving this problem is also an important direction for future work. For application, we will further learn a symbolic policy for automated CPU design based on this framework to optimize the performance/power/area (PPA) of the CPU.
This work is partially supported by the National Key R&D Program of China (under Grant 2021ZD0110102), the NSF of China (under Grants 61925208, 62102399, 62222214, 62002338, U22A2028, U19B2019), CAS Project for Young Scientists in Basic Research (YSBR-029), Youth Innovation Promotion Association CAS and Xplore Prize.
[1] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nat., 529(7587):484-489, 2016.
[2] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. J. Mach. Learn. Res., 17:39:1-39:40, 2016.
[3] Jiri Kubalík, Eduard Alibekov, and Robert Babuška. Optimal control via reinforcement learning with symbolic policy approximation. IFAC-PapersOnLine, 50(1):4162-4167, 2017. 20th IFAC World Congress.
[4] Daniel Hein, Steffen Udluft, and Thomas A. Runkler. Interpretable policies for reinforcement learning by genetic programming. Eng. Appl. Artif. Intell., 76:158-169, 2018.
[5] Mikel Landajuela Larma, Brenden K. Petersen, Sookyung Kim, Cláudio P. Santiago, Ruben Glatt, T. Nathan Mundhenk, Jacob F. Pettit, and Daniel Faissol. Discovering symbolic policies with deep reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5979-5989. PMLR, 2021.
[6] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel.  $r l^{2}$ : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016.
[7] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.
[8] Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J Smola. Meta-q-learning, arXiv preprint arXiv:1910.00125, 2019.
[9] Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. arXiv preprint arXiv:2107.02729, 2021.
[10] Miles D. Cranmer, Alvaro Sanchez-Gonzalez, Peter W. Battaglia, Rui Xu, Kyle Cranmer, David N. Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. CoRR, abs/2006.11287, 2020.
[11] Shuyao Cheng, Pengwei Jin, Qi Guo, Zidong Du, Rui Zhang, Yunhao Tian, Xing Hu, Yongwei Zhao, Yifan Hao, Xiangtao Guan, Husheng Han, Zhengyue Zhao, Ximing Liu, Ling Li, Xishan Zhang, Yuejie Chu, Weilong Mao, Tianshi Chen, and Yunji Chen. Pushing the limits of machine design: Automated cpu design with ai, 2023.
[12] John R. Koza. Genetic programming - on the programming of computers by means of natural selection. Complex adaptive systems. MIT Press, 1993.
[13] Michael D. Schmidt and Hod Lipson. Age-fitness pareto optimization. In Martin Pelikan and Jürgen Branke, editors, Genetic and Evolutionary Computation Conference, GECCO 2010, Proceedings, Portland, Oregon, USA, July 7-11, 2010, pages 543-544. ACM, 2010.
[58] Andrew Howard, Ruoming Pang, Hartwig Adam, Quoc V. Le, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Searching for mobilenetv3. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 1314-1324. IEEE, 2019.
[59] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[60] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. CoRR, abs/1806.09055, 2018.
[61] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[62] Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Li Fei-Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 82-92. Computer Vision Foundation / IEEE, 2019.
[63] Alexander L. Gaunt, Marc Brockschmidt, Rishabh Singh, Nate Kushman, Pushmeet Kohli, Jonathan Taylor, and Daniel Tarlow. Terpret: A probabilistic programming language for program induction. CoRR, abs/1608.04428, 2016.
[64] Matko Bosnjak, Tim Rocktäschel, Jason Naradowsky, and Sebastian Riedel. Programming with a differentiable forth interpreter. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 547-556. PMLR, 2017.
[65] Bernhard Wymann, Eric Espie, Christophe Guionneau, Christos Dimitrakakis, Rémi Coulom, and Andrew Sumner. Torcs, the open racing car simulator. Software available at http://torcs.sourceforge.net, 4(6):2, 2000.
A Symbolic Operators
To ensure the numerical stability of the proposed symbolic learning framework, we regularize the operators and employ a penalty term to keep the input from the "forbidden" area. We show the operators and the corresponding penalty terms as follows:
- Multiplying operator:
$$
y = \min  (m a x (x _ {1}, - 1 0 0), 1 0 0) * \min  (m a x (x _ {2}, - 1 0 0), 1 0 0),
$$
the penalty term can be formulated as:
$$
L _ {m u l} = \max  (x _ {1} - 1 0 0, 0) + \max  (- 1 0 0 - x _ {1}, 0) + \max  (x _ {2} - 1 0 0, 0) + \max  (- 1 0 0 - x _ {2}, 0)
$$
- Division operator:
$$
y = \left\{ \begin{array}{c l} 0, & x _ {2} <   0. 0 1, \\ \frac {x _ {1}}{x _ {2}}, & x _ {2} \geq 0. 0 1. \end{array} \right.
$$
$$
L _ {d i v} = \max  (0. 0 1 - x _ {2}, 0)
$$
- Sine operator:  $y = \sin(x)$ , the penalty term is set as zero:  $L_{\sin} = 0$ .
- Cosine operator:  $y = \cos(x)$ , the penalty term is set as zero:  $L_{\cos} = 0$ .
- Exponential operator:
$$
y = \exp (\min (\max (x, - 1 0), 4)),
$$
$$
L _ {e x p} = \max  (x - 4, 0) + \max  (- 1 0 - x, 0)
$$
- Log operator:
$$
y = \log (m a x (x, 0. 0 0 1)),
$$
$$
L _ {l o g} = \max  (0. 0 0 1 - x, 0)
$$
- Identity operators:  $y = x$ , the penalty term is set as zero:  $L_{identity} = 0$ .
- Condition operator:  $y = \text{sigmoid}(x_1) * x_2 + (1 - \text{sigmoid}(x_1)) * x_3$ , the penalty ter is set as zero:  $L_{\text{condition}} = 0$ .
In practice, the identity operator is only used in the plain structure. For all the environments in single-task RL and meta-RL, we use the symbolic network structure described in Section 3.1. Especially, for the meta-RL environment Cheetah-vel-ood, we add one Condition operator in each layer. We think the Condition operator will be useful for environments where the reward function changes. During training, we involve a penalty loss function  $\mathcal{L}_{\text{penalty}}$  which is the sum of the penalty terms of regularized operators in symbolic networks:
$$
\mathcal {L} _ {\text {p e n a l t y}} = \sum_ {i = 1} ^ {i = N} \mathcal {L} _ {g _ {i}} \left(x _ {i}\right). \tag {6}
$$
We show the learning curves of the penalty loss function for single-task RL in Figure 5 and for meta-RL in Figure 6. During the training process, for all environments in both single-task RL and meta-RL, the penalty loss function remains on a very small order of magnitude, which indicates that most of the operators in the symbolic network work the same as the original unregularized operators.
Figure 5: Learning curves of the penalty loss function in single-task RL. The shaded area spans one standard deviation.
Figure 6: Learning curves of the penalty loss function in meta-RL.
B Environment details.
B.1 Single-task RL
For single-task RL, we use the same benchmark control tasks as DSP [5]. We describe these environments as follows: