SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we carry out experiments by abating the features of CSP. We change the structure of the symbolic network and replaced the path selector with the  $L_{1}$  norm minimization. We compare the learning curves of the test task rewards in Figure 9. The results show that the dense connections effectively improve the performance and we can facilitate the search for the proper symbolic form by arranging the operators to further improve the performance. Besides, the path selector is better at selecting the symbolic policy from the symbolic network than  $L_{1}$  norm minimization.
F.3 Human Study for Interpretability
We also carry out a human study to evaluate the interpretability of the symbolic policies generated by ESPL. The assessment of policy interpretability actually requires some understanding of the environments and what each state variable means. We invited ten researchers to rate the interpretability of policies, with a maximum of 20 minutes for each policy. On a five-point scale, we told them that
the interpretability could be judged based on whether they could see from the policy expression what the policies was based on to make decisions and the possible corresponding relationships between actions and states. A score of five indicated that the strategy was highly interpretable and could be designed by humans, while a score of zero indicated that it was completely uninterpretable, just like the neural network policies. We measured the average score of the interpretability of the policies. We give the interpretability scores for ESPL and DSP in Table 16.
Table 16: Comparison of interpretability scores of ESPL and DSP.
CartPole	MountainCar	Pendulum	InvDoublePend	InvPendSwingup	LunarLander	Hopper	BipedalWalker
ESPL	5	5	4.5	4.1	4.3	3.9	2.6	3.1
DSP	5	5	4.5	5.0	4.2	4.0	3.1	3.2
Based on the results of the human study, we found that when the symbolic expressions were relatively short, there was a consensus that the expressions were highly interpretable. When the expression length became longer, the interpretability score decreased. It may be relatively difficult to understand symbolic expressions in a short time, but it is generally believed that interpretability is much higher than a black box, at least they can understand the action will be affected by which state, and the partial correlation. Besides, the symbolic policies produced by ESPL and DSP have comparable interpretability.
F.4 Analysis of symbolic policies for Meta-RL
We then analyze the symbolic policies for different tasks produced by CSP. For each environment, we sample 10 tasks from the environment task distribution and obtain the corresponding symbolic policies with CSP. Then we analyze the selected paths of these policies which determine the forms of the symbolic expressions. Table 12 shows the results. We calculate the average count of selected paths per action dimension among the policies<sup>5</sup>. We find that this number varies across different environments. The symbolic expression can be extremely short for simple environments like Cartpole or relatively long for complex environments like Walker2D. We also calculate the average count of paths that are selected by more than ninety percent of the symbolic policies. In almost all environments, the mostly selected paths account for a high percentage of the selected paths, which indicates that the expressions of symbolic policies for different tasks of the same environment share similar forms.
F.5 Theoretical Analysis for Cartpole
The theoretical analysis is as follows: The dynamic of cartpole system can be defined as:
$$
\ddot {x} = \frac {8 f a + 2 m \sin \theta (4 L \dot {\theta} ^ {2} - 3 g \cos \theta)}{8 M - 3 m \cos 2 \theta + 5 m}.
$$
$$
\ddot {\theta} = \frac {g \sin \theta - \frac {\cos \theta (f a + L m \dot {\theta} ^ {2} \sin \theta)}{m + M}}{L (\frac {4}{3} - \frac {m \cos^ {2} \theta}{m + M})}.
$$
Where  $f$  is the coefficient of force,  $a$  represents the action,  $m$  is the weight of the pole,  $M$  is the weight of the cart,  $L$  is the half-pole length,  $\theta$  is the angle between the pole and the vertical direction, and  $x$  denotes the horizontal coordinate of the cart.
$$
\text {D e f i n e} X = \left[ \begin{array}{c} x \\ \dot {x} \\ \theta \\ \dot {\theta} \end{array} \right], \text {t h e n} \dot {X} = \left[ \begin{array}{c} \dot {x} \\ \frac {8 f a + 2 m \sin \theta (4 L \dot {\theta} ^ {2} - 3 g \cos \theta)}{8 M - 3 m \cos 2 \theta + 5 m} \\ \dot {\theta} \\ \frac {g \sin \theta - \frac {\cos \theta (f a + L m \dot {\theta} ^ {2} \sin \theta)}{m + M}}{L (\frac {4}{3} - \frac {m \cos^ {2} \theta}{m + M})} \end{array} \right].
$$
<sup>5</sup>We only consider paths that contribute to the final expression. Besides, the count of the remaining paths may not equal to the count of operators in the final symbolic policy because some operators can be merged after simplifying symbolic expressions.
According to the Hartman-Grobman theorem, the local stability of this nonlinear system near its equilibrium point is equivalent to the linearized system near the equilibrium point. For cartpole system, the equilibrium point is  $[x,\dot{x},\theta ,\dot{\theta} ] = [0,0,0,0]$
If  $a = 0$ , the system can be linearized as:
$$
\dot {X} = \left[ \begin{array}{c c c c} 0 & 1 & 0 & 0 \\ 0 & 0 & \frac {- 6 g m}{8 M + 2 m} & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & \frac {g}{L (\frac {4}{3} - \frac {M}{m + M})} & 0 \end{array} \right] \left[ \begin{array}{c} x \\ \dot {x} \\ \theta \\ \dot {\theta} \end{array} \right].
$$
Calculate its eigenvalues:
$$
[ 0, 0, 3. 9 7 1 1 4 5 9 3, - 3. 9 7 1 1 4 5 9 3 ].
$$
Due to the presence of positive eigenvalues, according to the Hartman-Grobman theorem, the system is unstable.
If  $a = 17.17\theta + 1.2\dot{\theta}$ , linearize the system near the equilibrium point:
$$
\dot {X} = \left[ \begin{array}{l l l l} 0 & 1 & 0 & 0 \\ 0 & 0 & \frac {1 3 7 . 3 6 f - 6 g m}{8 M + 2 m} & \frac {9 . 6 f}{8 M + 2 m} \\ 0 & 0 & 0 & 1 \\ 0 & 0 & \frac {g - \frac {1 7 . 1 7 f}{m + M}}{L (\frac {4}{3} - \frac {M}{m + M})} & \frac {- 1 . 2 f}{L (m + M) (\frac {4}{3} - \frac {M}{m + M})} \end{array} \right] \left[ \begin{array}{l} x \\ \dot {x} \\ \theta \\ \dot {\theta} \end{array} \right]
$$
$$
[ 0 + 0. j, 0 + 0. j, - 2 6. 3 4 + 6. 6 5 0 1 4 2 8 6 j, - 2 6. 3 4 - 6. 6 5 0 1 4 2 8 6 j ]
$$
Since all the real parts of the eigenvalues are non-positive, according to the Hartman-Grobman theorem, the system is stable. Therefore, for the CartPole environment, the policies learned through ESPL can maintain the stability of the CartPole system.