Contextual Symbolic Policy For Meta-Reinforcement Learning
Authors: Jiaming Guo
Rui Zhang
Shaohui Peng
Qi Yi
Xing Hu
Ruizhi Chen
Kehan Long
Zidong Du
Xishan Zhang
Ling Li
Qi Guo
Yunji Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: meta learning
reinforcement learning
context variables
symbolic policy

Abstract: Context-based Meta-Reinforcement Learning (Meta-RL), which conditions the RL agent on the context variables, is a powerful method for learning a generalizable agent. 
Current context-based Meta-RL methods often construct their contextual policy with a neural network (NN) and directly take the context variables as a part of the input. However, the NN-based policy contains tremendous parameters which possibly result in overfitting, the difficulty of deployment and poor interpretability. 
To improve the generation ability, efficiency and interpretability, we propose a novel Contextual Symbolic Policy (CSP) framework, which generates contextual policy with a symbolic form based on the context variables for unseen tasks in meta-RL. Our key insight is that the symbolic expression is capable of capturing complex relationships by composing various operators and has a compact form that helps strip out irrelevant information. Thus, the CSP learns to produce symbolic policy for meta-RL tasks and extract the essential common knowledge to achieve higher generalization ability. Besides, the symbolic policies with a compact form are efficient to be deployed and easier to understand.
In the implementation, we construct CSP as a gradient-based framework to learn the symbolic policy from scratch in an end-to-end and differentiable way. The symbolic policy is represented by a symbolic network composed of various symbolic operators. We also employ a path selector to decide the proper symbolic form of the policy and a parameter generator to produce the coefficients of the symbolic policy. Empirically, we evaluate the proposed CSP method on several Meta-RL tasks and demonstrate that the contextual symbolic policy achieves higher performance and efficiency and shows the potential to be interpretable.

Under review as a conference paper at ICLR 2023
CONTEXTUAL SYMBOLIC POLICY REINFORCEMENT LEARNING
FOR META-
Anonymous authors Paper under double-blind review
ABSTRACT
Context-based ReitfoReifforemmentt Learning (Meta-RL), which conditions the RL agent on the context variables, a powerful method for learning general- izable agent. Current context-based Meta-RL RL methods often construct their con- textual policy with a neural network (NN) and directly take the context variables part of the input. However, the NN-based policy contains tremendous param- eters which possibly result in overfitting, the difficulty of deployment and poor interpretability. To improve the generation ability, efficiency and interpretability. we propose a novel Contextual Symbolic Policy (CSP) framework, which gen- erates contextual policy with a symbolic form based on the context variables for unseen tasks in meta-RL. Our key insight is that the symbolic expression is ca- pable of capturing complex relationships by composing various operators and has a compact form that helps strip out irrelevant information. Thus, the CSP learns to produce symbolic policy for meta- RL tasks and extract the essential common knowledge to achieve higher generalization ability. Besides, the symbolic poli- cies with a compact form are efficient to be deployed and easier to understand. In the implementation, symbolic policy from scratch in an end-to-end and differentiable way. The sym- bolic policy is represented by symbolic network composed of various symbolic operators. We also employ path selector to decide the proper symbolic form of the policy and a parameter generator to produce the coefficients of the symbolic policy. Empirically, we evaluate the proposed CSP method on several Meta-RL RL tasks and demonstrate that the contextual symbolic policy achieves higher perfor- mance and efficiency and shows the potential to be interpretable.
construct CSP gradient- gradient-based framework to learn the
INTRODUCTION
Retta-Reiffrremeentt Learning (Meta-RL) promising strategy to improve the generalization abil- ity on unseen tasks of reinforcement learning. Meta-RL methods learn the shared internal structure of tasks from the experiences collected across distribution of training tasks and then quickly adapt to a new task with a small amount of experiences. On this basis, context- tantext-based Meta-RL meth- ods (Duan et al. 2016; Rakelly al., 2019: Fakoor et al., 2019 Huang et al. [02I]) are proposed with the motivation that only a part of the model parameters need to be updated in a new environ- ment. They force their model to be conditional on set of task-specific parameters named context variables which are formed by aggregating experiences. Context- Context-based Meta-RL. methods are attrac- tive because of their empirically higher performance and higher efficiency compared with previous methods which update the whole model.
However, how to incorporate the context variables into the policy is still an open problem. Most of the current methods construct their contextual policy with neural network (NN) and directly take the context variables as a part of the input. This kind NN-based policy usually involves thousands of parameters, which may bring training difficulties, possibly result in overfitting and hurt the generalization performance. In addition, deploying the complex NN-based policy is inefficient and even impossible limited computational resources. What is worse, we have to treat the NN- based policy as a black box that is hard to comprehend and interpret, e.g., we cannot understand what the difference between the policies of different tasks is.

---

Under review as a conference paper at ICLR 2023
To address the above issues, in this work, propose novel Contextual Symbolic Policy (CSP) framework to learn a contextual policy with a compact symbolic form for unseen tasks in meta-RLe We inspired by the symbolic expression, which has compact form but is capable of capturing complex relationships by composing variables, constants and various mathematical operators. In general, compact and effective representations can strip out irrelevant information and find the sential relationship of variables, which can benefit the generalization. Therefore, for meta-RL tasks with similar internal structures, CSP produces symbolic policies model the relationship of the proper action and state and extract essential common knowledge across the tasks. With the com- mon knowledge of a series of tasks, CSP able to achieve higher generalization ability and quickly adapt to unseen tasks. Moreover, the compact symbolic policies learned by CSP are efficient to be deployed and easier to understand. In conclusion, contextual policies produced by CSP achieve higher generalization performance, efficiency, and show the potential to be interpretable when con- strained in a compact symbolic form.
However, finding the proper forms and constant values of the symbolic policies for distribution of tasks is challenging. In this paper, we propose an efficient gradient-basee learning method for the CSP framework to learn the contextual symbolic policy from scratch in an end-to-end differentiable way. To express the policy in a symbolic form, the proposed CSP consists of a symbolic network, path selector and parameter generator. The symbolic network can be considered the candidate symbolic policies. In the symbolic network, the activation functions are composed of various symbolic operators and the parameters can be regarded as the coefficients in the symbolic expression. For new task, the path selector chooses the proper compact symbolic form from the symbolic network by adaptively masking out most irrelevant connections. Meanwhile, the param- eters of the chosen symbolic form are generated by the parameter generator. We design all these modules be differentiable. Thus, we can update the whole framework with gradient efficiently.
full set of
We evaluate the proposed CSP on several Meta-RL tasks. The results show that our CSP achieves higher generalization performance than previous methods while reducing the floating point oper- ations (FI.Pss) by 2-45000x. Besides, the produced symbolic policies show the potential to be interpretable.
RELATED WORKS
2.1 META-REINFORCEMENT LEARNING
Bengio et al. 1991 Thrun & Prati 1998) to the context of reinforcement learning. Some works (Li et al. 2017: Young et al., 2018 Kirsch et al. 2019 Zheng et al., 2018 Sung et al. 2017 Houthooft et al. 2018) aim to meta-
Meta-RL extends the notion of meta-learning (Schmidhuber) 1987
learn the update rule for reinforcement learning. We here consider another research line of works that meta-train policy that can be adapted efficiently to a new task. Several works (Finn et al. 2017; Rothfuss et al. 2018 Stadie et al. 2018 Gupta et al. 2018 Liu et al., 2019) learn an initialization and adapt the parameters with policy gradient methods. However, these methods inefficient be- cause of the on-policy learning process and the gradient-based updating during adaptation. Recently, context-based Meta-RL achieve higher efficiency and performance. PEARL (Rakelly et al., 2019) proposes an off-policy Meta-RL method that infers probabilistic context variables with experiences from new environments. ADARL (Huang et al. (02I)) characterizes a compact representation about changes of environments with structural environment model, which enables efficient adaptation. Hyper (Sarafian et al., (02T) proposes a hypernetwork where the primary network determines the weights of a conditional network and achieves higher performance. Fu et al. (0220) introduce contrastive learning method to train a compact context encoder. They also train an exploration pol- icy to maximize the information gain. Most of the existing context-based Meta-RL methods(Fu et al. 2020 Zhang et al., 2021 Zintgraf et al. 2020) attempt to achieve higher performance by im- proving the context encoder or the exploration strategy. However, in this paper, we aim to improve the efficiency, interpretability and performance by replacing the pure neural network policy with a contextual symbolic policy.
2

---

Under review as a conference paper at ICLR 2023
2.2 SYMBOLIC REGRESSION
Symbolic regression aims find symbolic expressions to best fit the dataset from an unknown fixed function. Although this problem is likely NP-hard in principle, several works attempt to solve it with heuristic algorithms. Koza (1993) introduce genetic programming (GP) to evolve the symbolic expressions and a series of following works (Schmidt & Lipson| 2010 Cava et al. 2019 Virgolin et al. 2019 de França & Aldeia 2021) expand the basic genetic programming method improve the performance, Recently, some methods involve deep learning for symbolic regression. AI Feynman (Udrescu & Tegmark] 2019) utilizes neural networks to discover hidden simplicity in the dataset and break harder problems into simpler ones. DSR (Petersen et al. (02T) train a recurrent neural network (RNN) with reinforcement learning to produce the symbolic expression. Differentiable symbolic regression methods (Martius & Lampert 2016; [Shooo et al. 2018) use neural network whose activation functions are symbolic operators as a symbolic expression and decrease the length of symbolic expressions with L1 regularization. However, the plain structure and L1 regularization may fail with complex problems. We also employ a symbolic network for the differentiability. The difference is that we propose a densely connected symbolic network and probabilistic path selector, which enable symbolic meta-policy learning. Besides, these methods are designed for regression of a fixed function while we aim to learn a meta-policy which is conditional on the context variables.
Recently, some works employ symbolic regression methods to obtain symbolic policies for effi- ciency and interpretability. Kubalik et al. (2017) and Hein et al. (2018) aim to approximate sym- bolic policy with genetic programming but require a given dynamics equations or a learned world model. DSP (Larma et al., (202)) employ a recurrent neural network to generate the symbolic pol- icy. They use the average returns of the symbolic policies as the reward signal and train the neural network with risk-seeking policy gradients. However, for environments with multidimensional ac- tion spaces, they need pre-trained neural network policy as the anchor model. Besides, in this framework, a single reward for reinforcement learning involves many environmental interactions, which is inefficient and makes it hard combine the symbolic policy with Meta-RL. Recently, some works (Bastani et al. 2018: Verma et al., 2018) attempt to distill an interpretable policy from pre-trained neural network policy but have problem of objective mismatch (Larma al., (021). T). Different from the methods talked above, we propose an efficient gradient-based based framework to ob- tain the symbolic policy without any pre-trained model. As far as we know, we are the first to learn a symbolic policy from scratch and use the symbolic policy for Meta-RL RL. There exist other works introduce the symbolic to reinforcement learning. Garnelo et al. (2016); d d'vvila Garcez et al. (2018) learn the symbolic representation for better interpretability. Lyu et al. (2019) introduces symbolic planning for efficiency and interpretability. In this paper, we focus on learning the symbolic policy for Meta-RI..
PRELIMINARIES
In the field of meta-reinforcemeen learning (Meta-RL), we consider distribution of tasks p(is) with each task K ~ p(ti) modeled as a Markov Decision Process(MDP). In common Meta-RL settings, tasks share similar structures but differ in the transition and/or reward function. Thus, we can describe a task & with the 6-tuple (S, A, PRY, Pa, 7)) In this setting, S C R" is a set of n-dimensional states, A Rm is a set of m-dimensional actions, Px S A the state transition probability distribution, Po : S + [0, 1] is the distribution over initial states, Tw : S x A - IR is the reward function, and 7 E (0, 1) is the per timestep discount factor. Following the setting of prior works (Rakelly et al., 2019: Fakoor et al., 2019). we assume there are M meta- training tasks " .M sampled from the training tasks distribution Ptrasn (k). For meta- testing, the tasks are sampled from the test tasks distribution Ptear (K). The two distributions are usually same most settings but can be different in out-of-distributionOOnn settings. We denote context eT {($1, 01, $1. r...... (ST, ae, S' pr)} as the collected experiences. For context- based Meta-RL. RL, agent encodes the context into a latent context variable & with a context encoder q(2(cr) and the policy is conditioned on the current state and the context variable 2. During adaptation, agent first collect experiences for a few episodes and then update the context variables and maximize the return with the contextual policy. The Meta-RL objective can be formulated as max Bxwp(()) [[Er wa [[(K,, n, {(x[er))]]; where Ri(ss म, ((2(eT)) denotes the expected episode return.
x + [0, is
3

---

Under review as a conference paper at ICLR 2023
Contert Symbolic Policy
Parameter Generator
Farameter
training
Informnce
State
cy ((s, a,,s, ril
Signold
contect encader
((ri))
Action
Score
Maek
Actor Loss
forward flow
@(s.
Critic Loss
Figure 1: Illustrations of framework. The Contextual Symbolic Policy (CSP) produces different symbolic policies based the context variables across tasks. CSP construct the symbolic policy with the symbolic network, selects the proper symbolic form with the Path Selector and generate the parameters symbolic policy with the Parameter generator. The whole framework is differentiable and can be learned end-to-end.
4 GRADIENT-BASED CONTEXTUAL SYMBOLIC POLICY
This section introduces the structure of our contextual symbolic policy, an end-to-end differentiable system that can be directly updated with gradient. As Figure shows, the contextual symbolic policy consists of three main components: 1) the Symbolic Network, which expresses the policy in symbolic form, 2) the Parameter Generator. which outputs the parameters for the symbolic network according to the context variables, and 3) the Path Selector, which select paths from the symbolic network to form compact symbolic expressions.
4.1 DENSELY CONNECTED S SYMBOLIC NETWORK
To construct symbolic policy in an end-to-end differentiable form, we propose the densely con- nected symbolic network. Inspired by previous differentiable symbolic regression methods (Martius) Lampert 2016 [ahoo et al., 2018), we employ neural network with specifically designed units, which named symbolic network. We now introduce the basic symbolic network named plain structure which illustrated Figure The symbolic network is a feed-forward network with layers. Different from traditional neural networks, the activation functions of the symbolic network replaced by symbolic operators, e.g. trigonometric functions and exponential functions. For the 4n layer of the symbolic network, denote the input 21-1 and the parameters wt, bj. These parameters serve as the constant in symbolic expression. We assume that the ltb. contains mn unary functions (ot,... > I.h and binary functions {oî 3 Firstly, the input of the 4sh layer will be linearly transformed by a fully-comnen colyccconnected layer:
Fi(x) = wiz + b.
The fully-connecten layer realizes the addition and subtraction in symbolic expressions and produces m 2n outputs. Then the outputs will go through the symbolic operators and be concatenated to form the layer output:
Gi(y) = (oi),,,... ......................... .9th ( (ym)), ). (ym+1 22) . (y 22- mm+2aa)
(2)
Then the 44 layer of the symbolic network can formulated as S Hj = G. (F(( Following the last layer, there will be fully-comme connecedd layer produce single output. For multiple action dimensions, we construct a symbolic network for each dimension of action.
Symbolic operator. The symbolic operators are selected from a library L. e.g. (sin, cos, eap, log, x)+} for continuous control tasks. For the plain structure, we include an identical operator which retains the output of the previous layer to the next layer in the library. We also provide optional conditional operator c( b, c) sigmoid(a) x xb+ (1-1111nmm) sigmoid(a)) to approximate the symbolic policy with gradient. Thus, it is critical ensure the numerical stability of the
then B else C. We find it is useful in some tasks. Note that we aim to find
4

---

Under review as a conference paper at ICLR 2023
Example network structures for the symbolic network. Left: the plain structure. Middle: symbolic work with dense connections. Right: a symbolic network with dense connections and arranged operators.
Figure 2:
system. However, this is not natural in a symbolic network. For example, the division operator and the logarithmic operator will create a pole when the input goes to zero and the exponential function may produce a large output. Thus, we regularize the operators and employ a penalty term keep the input from "forbidden" area. For example, the logarithmic operator y log(x) returns (x)(x) for x > boundiog and log (booudddg)))) otherwise and the penalty term is defined as Ling = maa (bounddog - x. 0). The division operator e = a/b returns a/b for b > bounddree and O otherwise. The penalty term is defined as Lair mar regularized operators can be found in the Appendix.
b, 0). The details of all
Dense connectivity. We introduce dense connections (Huang et al. 2017) in the symbolic net- work, where inputs of each layer connected to all subsequent layers. Consequently, the lea layer of the symbolic network will receive the environment state and the output of all preceding layers " 21 Gr(Fi(ss, (1, .I)) On the one hand, the dense connections improve the information flow between layers and benefit the training procedure. On the other hand, the dense skip connections across layers enable us to control the complexity of the symbolic expressions with the parameters more flexibly. In practice, each layer in the symbolic network may contain different operators. With the dense connectivity, we can flexibly arrange the position of operators. For ex- ample, if we only arrange the sin operator in the last layer but the oracle expression contains terms like (5)). the input of the sin operator can still be from the original state because of the dense connections. We give example of arranged operators in Figure 2 which we use for all tasks in the experiments. In this symbolic network, we assume that multiplication and division operations more likely to occur at shallow layers, while more complex operations such as sines and cosines are more likely to occur at deep layers. In addition, we avoid the form sin(-.- crp(...) ...) which rarely occurs in physics formulas with the arrangement.
4.2 INCORPORATING THE CONTEXT VARIABLES
To produce different symbolic policies with the symbolic network for different tasks x sampled from the task distribution p(k), we need incorporate the context variables ((2(or) to the symbolic network. To condition the parameters of the symbolic expression on the context variable, we propose parameter generator: wg = (=z) which is a neural network produce the parameters of symbolic networks for all action dimensions based on the context variables.
However, the symbolic network serves select the proper paths from the symbolic network to produce a compact symbolic policy, we reduce the number of paths involved in the final symbolic policy then proper paths remain and redundant paths are removed. This can be naturally realized by minimizing the Lp norm of the symbolic network parameters. As the Log norm is not differentiable, some methods (Martius & Lampert 2016) Sahoo et al. 2018) employ L2 norm instead of Lo norm. However, L1 will penalize the magnitude of the parameters. In our framework, the parameters of the symbolic network the output a neural network rather than independent variables. The penalty of magnitude may severely affect the training of the parameter generator. Inspired by the probability- based sparsification method (rriniass al. 2017] Louizos et al., 2017, Zhou et al. 2021), we propose probabilistic path selector which selects path from the network by multiplying a binary mask on the parameters of the symbolic network. The path selector first produce scores with the context variables: v(=),
full set of the search space of symbolic expressions. To
5

---

Under review as a conference paper at ICLR 2023
where 8; E (0, 1). The score 8, serves the probability of the Bernoulli distribution and the binary mask m, is sampled from the distribution: my Bern(s;). Then the final parameters of the symbolic network Consequently, to get a compact symbolic expression, we only need to minimize the expectation of the Lo norm of the binary mask Em->p(maa) |m Σ 8is without penalizing the magnitude of the parameters.
ug X m, where X is the element-wise multiply operation.
During the process of collecting data or testing, we can directly sample the binary mask from the Bernoulli distribution. However, the sampling process does not have a well-defined gradient. Thus, for the training process we build up our sampling function with the gumbel-softmax trick (Jang et al. 2016). As the mask m is binary categorical variables, we replace the softmax with sigmoid and named the sampling function as gumbel sigmoid. The gumbel sigmoid function can be formulated as:
4 log I-8 + 91 90
Mgs sigmoid(
(3)
T
where 91 and go are i.i samples drawn from Gumble(0, 1). T is the temperature annealing pa- rameter. Note that mgs is still not binary mask. To obtain a binary mask but maintain the gradient, we employ the Straight-Throus (2).5(x) E 13 is the indicator function and the overline means stopping the gradient. With the path selector, the framework is able to produce very short symbolic policies for relatively simple tasks while produce complex but compact symbolic policies to handle hard tasks like Walker2d and Hopper in Mujoco SimulatorTTodorovv
(ST) trick: m 110.5 + mgs mgs- where
Todorov et al. 2012).
5 META-LEARNING THE SYMBOLIC POLICY
In this section, we introduce the meta-learning process of our symbolic policy. Following PEARL (Rakelly et al. 2019). build up off- off-policy learning framework on top of the soft actor-critic al- gorithm (SAC) (Haarnoja et al., 2018). The main differences are the additional loss for the symbolic policy and the schedule for collecting data and simplifying symbolic expressions.
5.1 Loss FUNCTION
We now illustrate our additional loss function for the symbolic policy. As described in Section 4.1 ensure the numerical stability, we regularize the symbolic operators and employ a penalty term for regularized operators. During training, we involve penalty loss function Lpennatty which is the sum of the penalty terms of regularized operators in symbolic networks:
t= t= #ij=LkN
Cpenatty (e (e,, 00) - 1000->1 -
Can.. (Ii...);
(4)
where 0g, Oy is the parameters of the parameter generator and the path selector, M is the dimension of action, L is the number of layers in symbolic network, N₁ is the number of regularized operators in layer J. Lij. is the input of operator I..j.k and Lg.,, is the penalty term for this operator. We also involve loss function Lectect to regularize the sum of score s which is the expectation of the Lo norm of the binary mask m as described in Section 4.2 To limit the minimum complexity of symbolic policies, we involve the target Lo norm defined as htrggt.. Then the loss function can be defined as:
Eselect(@g) max Σ 8 Hargget 0)
(5)
5.2 TRAINING SCHEDULE
In practice, we train our symbolic policy in an off-policy manner. For meta-training epoch f, the agent first collects experiences into the corresponding buffer B,, for several iterations. At the beginning of each iteration, we sample context eT from buffer Bx, and sample context variables 2 ~ q(2 as PEARL does. The difference is that we also sample the symbolic policy with ((2) and (2) and use the sampled policy for the following steps of the iteration. Then sample RL batch and context from the buffer and optimize the context encoder q(2(or) to recover the state- action value function. For each training step, sample symbolic policy. We employ the

---

Under review as a conference paper at ICLR 2023
Hoppar-params
cheetah-vel.com
return
return
Hyper
enV slegs
BBV slegs
BIV sleps Campooe---ood
Conarlander-parraee
InvDoublePenddpari
retum
return
stegs
Steps
Figure 3: mean and standard deviation returns on test tasks averaged over five runs.
Comparison for different kinds of contextual policies Meta-RL tasks. We show the
soft actor-critic to optimize the state-action value function. For the parameter generator and the path selector, we employ Leelect and Cpenatty in addition the SAC loss. During training, decrease the temperature parameter T of gumbel sigmoid linearly and decrease the kaarge from the length of the origin parameters Wg to a target value with parabolic function. We illustrate the gradient flow in Figure I More details and the pseudo-code can be found in the Appendix.
EXPERIMENT
In this section, we present the experimental results of our contextual symbolic policy (CSP). We first compare CSP to prior meta policies on several Meta-RL problems to evaluate the performance and the inference efficiency Section Then we analyze the symbolic policy generated for different Finally, carry out ablation experiments for CSP and show the results in 6.3
tasks in Section 6.2
6.1 COMPARISON OF CONTEXTUAL POLICIES
Experimental Settings. We first evaluate CSP on several continuous control environments. We modify the environments of OpenAl Gym (Brockman et al. 2016), including kinds of classic con- trol, Box2D,, and MuloCo (Todorov et al. 2012) to be Meta-Rí. tasks similar to Rakelly et al.) (2099) [Hang et al. (20TT); et al. (020). These environments require the agent to adapt across dynamics (random system parameters for Hopper-params, Walker2d-parame Lumarlander-parant, Inv inyDoublePendd-araal
different force magnitude and pole length for Cartpole-fl-ood) or reward functions (target velocity for Cheetah-vel-ood), For the symbolic network, we use the arranged densely connected structure in Figure 2 We run all environments based the off-policy meta- learning framework proposed by PEARL and use the same evaluation settings. We compare CSP with PEARL. which concatenates the observation and context variables as the input of policy and Hyper (Sarafian et al., 202T) which generate the parameters of policy with a ResNet model based on the context variables. Note that the original Hyper also modifies the critic, but we build all the critics with the same network structure for consistency. More details of the environments and hyper-parametes can be found in the Appendix.
Performance comparisons. In Figure 3 we report the learning curves of undiscounted returns on the test tasks. We find that in all the environments, CSP achieves better or comparable perfor- mance than previous methods. In Hopper-params, Walker2d-params and muvDDublePend-paraas,, CSP outperforms PEARL and Hyper during the whole training process. In Lunarlander-params. CSP achieves better final results. In Cartpole-floow CSP adapts to the optimal more quickly. In the out-of-distribution task chootah-vel-ood, during training because of over- wittiigg But our CSP is less affected. In conclusion, expressing the policy in the symbolic form helps improve the generalization performance.
find the performance of PEARL and Hyper decrease

---

Under review as a conference paper at ICLR 2023
Table 1: FLOPS and inference time of different contextual policies.
FLOPs/k
Times/ms
Environment
CSP
PEARL
Hyper
CSP
PEARL
Hyper
Walker2d-paraan params
3.11
189.31
5.64
20.89
27.00
22.64
Hopper-parane
0.51
186.90
4.10
4.13
26.58
17.23
Dovllelleeendd paramps
0.039
186.00
3.59
0.37
25.05
12.32
Cartpole-fl-oo fl-ood
0.004
183.90
1.79
0.042
23.90
9.08
Lunarlander-g
0.015
185.4
3.08
0.14
23.43
12.34
Cheetah-vel-ood
0.53
190.21
7.18
4.90
28.44
24.16
Table 2: Average count of all selected paths and paths selected by at least ninety percent policies.
Environment
Selected paths
Mostly selected paths
Walker2d-params
76.42
70.3
Hopper-params
21.5
20.33
wwDDoublcPeddparamme
23.2
21.0
Cartpole-fl-ood
3.06
3.0
Lunarlander-g
5.2
5.0
Cheetah-vel-ood
27.04
18.5
Efficiency comparisons. We also evaluate the deploying efficiency of contextual policies. We first calculate the flops of each kind of policy per inference step. Then we consider application scenario that the algorithm control five thousand simulated robots with the Intel(R) Xeon(R) Gold 5218R @ 2.10GHz CPU and record the elapsed time per inference step [
We report the results in Table T Compared to PEARL, CSP reduces the FLOPs by 60-45000x and reduces the inference time by up to 600x. Compared to Hyper, CSP reduces the flops by 2-450x and reduces the inference time by up to 200x. Thus, compared with pure neural network policies, the contextual symbolic policy has a significant advantage in computational efficiency.
6.2 ANALYSIS OF SYMBOLIC POLICIES
We then analyze the symbolic policies for different tasks produced by CSP. For each environment, we sample 10 tasks from the environment task distribution and obtain the corresponding symbolic policies with CSP. Then analyze the selected paths of these policies which determine the forms of the symbolic expressions. Table 2 shows the results. We calculate the average count of selected paths per action dimension among the policies We find that this number varies across different vironments. The symbolic expression can be extremely short for simple environments like Cartpole or relatively long for complex environments like Walker2D. We also calculate the average count of paths which are selected by more than ninety percent of the symbolic policies. In almost all environ- ments, the mostly selected paths account for high percentage of the selected paths, which indicates that the expressions of symbolic policies for different tasks of the same environment share similar forms.
The proposed CSP can also improve interpretability. We take the Cartpole-fi-ood environment as an example and illustrate the Cartpole system in Figure + The form of the symbolic policies produced by CSP is action = cl + 02 * + 6, where is the the angle of the pole and is the rotation rate of the pole. cl and c2 are the positive coefficients and b is small constant which can be ignored. The action is the force scale to push the cart. Then the policy can be interpreted as pushing the cart in the direction that the pole is deflected will be deflected. To analyze the difference between policies for different tasks, we uniformly set the force magnitude and the length of the pole. Then generate the symbolic policy with CSP and record the coefficients. As Figure tend to increase when the force magnitude decrease and the length increase, which is in accord with our intuition. We will give examples of symbolic policy for other environments the Appendix.
el and c2
Note we accelerate Hyper by only updating the parameters of policy when we update the context variables. Thus, Hyper only uses the policy model without the ResNet model to infer an action. We only consider paths that contribute the final expression.
8

---

Under review as a conference paper at ICLR 2023
2
15
6.7
0.5
0.6
Figure 4: The Cartpole system be Figure 5: The coefficients of symbolic policies for Cartpole environ- controlled.
ments with different force magnitude and pole length.
whate
retum
return
retum
CSP ESP
CSP RELU ESP TANH
env steps
CITV Steps
eTV Steps
Figure 6: Ablation results of the symbolic network structure (left), the path selector(middle) and the symbolic operator(right). CSP means the plain structure and CSP means the densely connected structure. CSPLLI means replacing the path selector with the L₃ norm minimization. CSP_TANE means replace all the symbolic operators with tanh. CSP RELU means replace all the symbolic operators with relu.
6.3 ABLATION
Finally, we carry out experiments by ablating the features of CSP. We first examine our choice the symbolic structure. We replace the symbolic network with plain structure and densely connected structure and compare the test task performance on Hopper-parane params environments. As Figure shows, the dense connections effectively improve the performance and we can facilitate the search for the proper symbolic form by arranging the operators to further improve the performance. We also replace our path selector with the L1 norm minimization. For the stability of training, we linearly increase the scale of the L, loss. To get a compact symbolic expression, we set the parameters with the absolute value less than 0.01 the Hopper-params and show the learning curves of return in Figure 6 Besides, we calculate the average Lo norm of the mask for our path selector and the count of non-zero parameters for L1 norm minimization which is 30.38 and 34.59, respectively. Compared with the L2 norm minimization, our path selector achieves higher performance when producing slightly more compact symbolic policies. We also replace the symbolic operators with commonly used activation functions tanh and relu. We use the same framework select the proper paths and set the same final L1 norm. The results in Figure [ show that by combine different operators to form the symbolic policy, CSP are able to handle complex relationship between action and state and achieve higher performance compared with single kind of operators.
the end of training. We carry out experiments
CONCLUSION
In this paper, we propose to learn a contextual symbolic policy for Meta-RL. In our gradient-based learning framework, we train the contextual symbolic policy efficiently without any pre-trained model. The contextual symbolic policy achieves higher generalization performance than previous methods. Besides, it is more efficient when deployed and has better interpretability. Our approach may inspire future works of symbolic policy for reinforcement learning or meta- meeta-reiforremment learn- ing.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. Advances in neural information processing systems, 31, 2018.
Y. Bengio, S. Bengio, and J. Cloutier. Learning synaptic learning rule. In IJCNN-91-Seatie International Joint Conference on Neural Newworks, volume ii, pp. 969 vol.2-, 1991. doi: 10. 19W.UCNN.1991.155000
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1606.06660.//444 2016.
William G. La Cava, Tilak Raj Singh, James Taggart, Srinivas Suri, and Jason H. Moore. Learn- ing concise representations for regression by evolving networks of trees. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
www.:nrumtttottotooooo....mutttt d=Hke- JhA9Y7
Artur S. d' Avila Garcez, Aimore Resende Riquetti Dutra, and Eduardo Alonso. Towards symbolic reinforcement learning with common sense. CoRR, abs/1804.08597, 2018. URL http://
arxiv.org/abs 1804. 08597
Fabrício Olivetti de França and Guilherme Seidyo Imai Aldeia. Interaction-trunsffrmattt evo- lutionary algorithm for symbolic regression. Evol. Comput., 29(3):367-390, 2021. doi: 10.1162/evco).aa..o00 .00285. htthttps:://// http://:///...../.....11.1111
http//...162//veoooaaooo 00285
Yan Duan, John Schulman, Xi Chen, Peter I. Bartlett, llya Sutskever, and Pieter Abbeel. rI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J Smola. Meta-q-learning. arXiv preprint 19Xxv099/0.0// 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference machine learning, pp. 1126-115. PMLR, 2017.
Haotian Fu, Hongyao Tang, Jianye Hao, Chen Chen, Xidong Feng. Dong Li, and Wulong Liu. Towards effective context for meta-reinforcement learning: an approach based on contrastive learning. arXiv preprint arXiv:2009. 13891, 2020.
Marta Garnelo, Kai Arulkumaran, and Murray Shanahan. Towards deep symbolic reinforcement learning. CoRR, abs/1609.05511, 2016. URL http:: org/ abs 100. 05518
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. Advances neural information pro- cessing systems, 31, 2018
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Daniel Hein, Steffen Udluft, and Thomas Runkler. Interpretable policies for reinforcement learn- ing by genetic programming. Eng. Appl. Artif. Intell., 76: 15-1158-666 169, 2018. doi: 10. 00.1016/j.enaappa. 2018.09.007. URL htttstt:::// /ttpa://ddoo.oe
http:ttttt/:/o.orrrttt htt:://oi..ogg///////// wrgg/1 1016
2018 http:///....11oo.eeappp.......a. enw.ppai.apppp........
007
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAI Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. Advances in Neural Information Processing Systems, 31, 2018.
Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. Adarl: What, where, and how to adapt in transfer reinforcement learning. arXiv preprint arXiv:2107.02729 2021.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
10

---

Under review as a conference paper at ICLR 2023
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softnax. arXiv preprint arXiv:1611.0114 2016.
Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving generalization in meta
reinforcement learning using learned objectives. arXiv preprint arXiv:1910.0400 04098, 2019.
John R. Koza. Genetic programming on the progranminng of computers by means of natural selection. Complex adaptive systems. MIT Press, 1993. ISBN 978-0-262-11170-6
Jiří Kubalík, Eduard Alibekov, and Robert Babuška. Optimal control via reinforcemett learning with 50(1))4162 01:4122-4167, 2017. ISSN 2405-8963. doi: http://cwooooroo/.o.oooffffeeeo. 805. URL ht https: // /www. seisssiienee enceedeeeee.... 20th IFAC World Congress.
symbolic policy approximation. IFAC- PapersOnnn,,
com/ htt::/ciinccceee http://ceeeeeeerrrreeee/////////
Mikel Landajuela Larma, Brenden K. Petersen, Sookyung Kim, Cláudio P. Santiago, Ruben Glatt, T. Nathan Mundhenk, Jacob F. Pettit, and Daniel Faissol. Discovering symbolic policies with deep reinforcement learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 5979-5989 PMLR, 2021. URL http: /ttt:://procedll ngs 1 r., wwwss/essvvve. /vtas//v13/// anda jue a21 html
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few- shot learning. arXiv preprint arXiv: 2017
Hao Liu, Richard Socher, and Caiming Xiong. Taming maml: Efficient unbiased meta-
reinforcement learning. In International conference on machine learning, pp. 4061-4071, PMLR, 2019.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint arXiv:1712.01322 2017.
Daoming Lyu, Fangkai Yang, Bo Liu, and Steven Gustafson. SDRL: interpretable and data- efficient deep reinforcement learning leveraging symbolic planning. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Ad- vances in Artificial Intelligence. EAAI 2019, Honolulu, Hawaii, USA, January 27 Febru- URL
ary 1, 2019, pp. 2970-2977, AAAI Press, 2019. doi: 10.
https://doii http///ii.ogg///// or0/10..0.000 609 aaai vv31011 03012973309970
Georg Martius and Christoph H Lampert. Extrapolation and learning equations. arXiv preprint arXiv:1610.02995. 2016.
Brenden K. Petersen, Mikel Landajuela Larma, T. Nathan Mundhenk, Cláudio Prata Santiago, Sookyung Kim, and Joanne Taery Kim. Deep symbolic regression: Recovering mathematical ex- pressions from data via risk-seeking policy gradients. In 9th International Conference on Learn- ing Representationss ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL BtLLhtttp:://ppeeee///
wet.:iiwwweeeeeeeeeee....ee.eeee hwtp:/eeeee/.ouueeeeeeee forum? hott:/ooooo.dddn00000001..
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinffrrcemett learning via probabilistic context variables. In International conference on machine learning, pp. 5331-5340. PMLR, 2019.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint /
2018.
Subham S. Sahoo, Christoph H. Lampert, and Georg Martius. Learning equations for extrapolation and control. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML. 2018, Stockholmsmüssan Stockholm, Sweden, July 10- 15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4439 4447. PMIR, 2018.
mmlr. wwwpsseeseveeeee.e v sahsoahllle html
Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypemneworks. In International Conference on Machine Learning, pp. 9301-9312. PMLR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Jurgen Schmidhuber. Evolutionary principles in self-referential learning- learning now to learn: The setta-mttammtt..-hooo Diploma thesis, Technische Universitat Munchen, Germany, 14 May 1987. URL http:// /tt::://wwwww.. iisia. ch/ wttp:/ssaaaaaeeeeeeeeeeeeeaaeaie wwwpeerenn//iploaaa....e. html
Michael D. Schmidt and Hod Lipson. Age-fitness pareto optimization. In Martin Pelikan and Jürgen Branke (eds.), Genetic and Evolutionary Computation Conference, GECCO 2010, Proceedings, Portland, Oregon, USA, July 7-11, 2010, pp. 543 544, ACM. 2010. doi: 10.1145/1830483. 1830584. URL https:// doi org. 111.1....1111111111111 18305844
Suraj Srinivas, Akshayvarun Subramanya, and R Venkatesh Babu. Training sparse neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 113--45, 2017.
Bradly Stadie, Ge Yang. Rein Houthooft, Peter Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. The importance of sampling inmeta-reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning learn: Meta-critic networks for sample efficient learning. arXiv preprint arXiv: 2017.
Sebastian Thrun and Lorien Pratt. Learning to learn. 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEF/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Malvian Udrescu and Max Tegmark. AI feynman: a physics-inspired method for symbolic regression. CoRR, abs/1905.. 1481, 2019. http::tttt//iiii httpaaiii...ggaa/a/a//ae wttp://pg/pp....o...o.
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. International Conference on Machine PMLR, 2018.
Learning, pp.
Marco Virgolin, Tanja Alderliesten, and Peter A. N. Bosman. Linear scaling with and within mantic backpropagation-baal genetic programming for symbolic regression. In Anne Auger and Thomas Stützle (eds.), Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2019, Prague, Czech Republic. July 13-17, 2019, pp. 1092-1092. ACM, 2019. doi: 10.1145/3321170..3321233 URLhttps:///oot wtt.://rrrr.1.11133222211111 332277
Kenny Young, Baoxiang Wang, and Matthew E Taylor. Metatrace actor-critic: Online step- size tuning by meta-gradient descent for reinforcement learning control. arXiv preprint 1rX1v:1805.04514. 2018
Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and Chongjie Zhang- Metacure: Meta reinforcement learning with empowerment-driven exploration. In International Conference on Machine Learning, pp. 12600-12610. PMLR, 2021.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances in Neural Information Processing Systems, 31, 2018
Xiao Zhou, Weizhong Zhang, Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. -608, 2021.
I Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep RL via meta- learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https::/
Forum?id=Hk 991lyYr]
12


Reviews:
[{"review_id": "p80j_Zexe1", "reviewer": "Reviewer_EqMi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To learn symbolic policies that generalize better while maintaining\ninterpretability, the authors of this paper propose to use neural\nnetworks to generate parameters of a symbolic network which is then\n\"pruned\" through a path selector. Both the generator and the path\nselector are conditioned on context variables in the setting of\ncontextual meta RL. The primary finding is that the learned policies can\nbe interpretable and generalize better to different contexts, such as\ndiffering mass and forces in a control problem.\n\n", "summary_of_the_review": "# Decision\n\nThe goal of this paper is to learn policies for meta-learning that are\nmore interpretable and more generalizable, This is a valuable goal to\nachieve in meta-RL and the research direction, using symbolic networks\nthat are generated by context, is interesting. While interpretability is\nsomewhat demonstrated, improvements in generalization are not addressed.\nIn addition, the symbolic network itself does not seem very different\nfrom a regular neural network that is arranged with special activation\nfunctions. The similarity to regular neural networks, as well as the use\nof regular neural networks to generate the symbolic policy, cast doubt\non the motivation and the broad effectiveness of symbolic networks.\nOverall, the research direction is not well-motivated and I find that\nthe paper further undermines the motivation. As a result, I am rating\nthe paper at a weak reject.", "strength_and_weaknesses": "# Strengths\n\n-   An interesting approach to learning interpretable policies. Using\n    both a generator and a path selector for learning a symbolic network\n    seems new, especially for contextual meta-RL\n\n# Weaknesses\n\n-   The main motivation is unclear. While the paper motivates the need\n    for symbolic networks by saying that overparameterized neural\n    networks can lead to overfitting and generalization, this is never\n    established and the current literature does not support that claim.\n    In addition, the way that the symbolic network is learned is, in the\n    end, with two neural networks. This seems to further undermine the\n    motivation.\n-   It is not that obvious how symbolic networks, as described in the\n    plain and densely-arranged architecture, differ from neural networks\n    aside from differing activation functions. If this is the only\n    difference, then this seems like a rather limited distinction from\n    regular neural networks. While in continuous control problems, your\n    choice of activation functions may lead to a more interpretable\n    policy - this may not be the case for some other problem for which\n    the needed non-linear activation function is not represented in your\n    library.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gKPTeMFPL3b", "reviewer": "Reviewer_KRNC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to parametrize meta-RL policies with a symbolic policy in the hopes of improving generalization, interpretability, and efficiency.", "summary_of_the_review": "Overall, this work does not meet or test the claims that it sets out and could significantly be improved in quality and presentation. Therefore, I do not think it currently is ready for publication.", "strength_and_weaknesses": "This work has two main weaknesses. First, the clarity of the writing and presentation could be significantly improved, and is currently challenging to understand. I elaborate upon that in the next box. Second, this work does not meet the main claims that it sets out of increasing generalization and interpretability. More specifically, the experimental evaluation of these items is not sufficiently thorough:\n- It is true that in the reported experiments, the baselines' performance degrades with further training on the \"OOD\" tasks, while the proposed method achieves high performance throughout training. However, early in training the baselines do still achieve near-optimal results on the OOD test set, and its further performance drop can easily be remedied with early-stopping, which makes this sort of generalization gap less significant. More significant to the community would be considering domains where existing methods cannot already easily generalize out of distribution.\n- The claims about interpretability of the system are unclear. It is somewhat interesting that the symbolic policy produces an equation that relates to physics on the cartpole domain, as reported in the experiments. However, can this only happen when there are no dense layers interleaved with the policy? If the \"symbolic\" operations occur in between dense layers, how can a tight analytic equation be extracted? Additionally, it is unclear to me that compositions of operations such as sin, cos, division, etc., are any more interpretable than large matrix multiplications, especially when such compositions are long. Finally, in image-based domains, such \"symbolic\" operations seem unlikely to be helpful anyway.\n- The experimental evaluation setup could generally be improved to be more informative. Specifically, existing works already evaluate on common MuJoCo benchmarks, and it seems useful to evaluate on those rather than creating new ones. It's unclear to me if the Hopper and Walker evaluations indeed are the ones that have been used in other works, or if they are variants created newly in this paper. If they are the common ones, my concern here is alleviated. Additionally, though, it's challenging to evaluate CSP's performance without comparison to other existing methods. For example, \"Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs\" reports strong performance of RL^2, and VariBAD also reports stronger-than-PEARL results. Inclusion of these methods would help understand whether CSP indeed offers superior performance or not.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WZRoSudDCC", "reviewer": "Reviewer_SCBW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Contextual Symbolic Policy learning for Meta-Reinforcement Learning, in which a symbolic network represents some search space in the policy space, conditioned on the context represented by encoding previous transitions using an encoder network. In addition, a path generator neural network is trained to select the \"correct\" symbolic form of the policy, by learning to mask out irrelevant nodes and edges in the symbolic policy network, with the encoded context as input. A parameter generator neural network is also trained, with the encoded context as input, to generate parameters of the symbolic policy network. In general, the whole pipeline, starts with design of the symbolic network architecture (plain, densely connected or arranged densely connected structures), learns to select the appropriate form of the symbolic policy by masking out irrelevant nodes and edges, and also learns the weights on the relevant edges. The pipeline is trained end-to-end.\n\nThe proposed approach is evaluated on a select benchmarks Meta-RL tasks and was shown to be better and sometimes competitive to the baselines.", "summary_of_the_review": "While the paper proposes an approach that allows extraction of symbolic policy expressions and has efficient inference speed, no analysis of the training efficiency is provided, especially given it has additional neural networks that need to be trained. Furthermore, it's not clear if policies of more complex environments will be any interpretable and if performance will scale to large benchmark problems. \n\nThe paper claims that by combining different kinds of operators to form the symbolic policy, CSP can handle more complex environments but this is not supported anywhere in the paper.", "strength_and_weaknesses": "Strengths\n\nThe learned symbolic policies are more interpretable than pure deep learning policies.\nThe learned symbolic polices have been demonstrated to be lightweight and easier to deploy.\n\nWeaknesses\n\nWhile the efficiency of inference is shown to be better, no training efficiency is demonstrated. The proposed approach contains two more neural networks to be trained.\nThe evaluation is only demonstrated on toy examples. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8NEuO0ejuK", "reviewer": "Reviewer_8Fo8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the issues in the neural network-based policy in meta reinforcement learning (RL), such as overfitting \\& poor generalization ability, difficult/inefficient to deploy with limited computational resources, and poor interpretability. To address those issues, the framework of Contextual Symbolic Policy (CSP) is proposed by learning a contextual policy with a symbolic form based on the context variables for unseen tasks in meta-RL. Finally, experiments were conducted on several continuous control problems, with results demonstrating its effectiveness in terms of return, FLOPs, and interpretability.\n", "summary_of_the_review": "According to my comments in both the main Weaknesses and the section of Clarity, Quality, Novelty, I feel this is a paper where reasons to reject outweigh reasons to accept.", "strength_and_weaknesses": "Strengths:\n- Symbolic representation is attractive since it has shown some promising in mitigating the issues regarding generalization ability, sample efficiency, and interpretability. This work targets to applying the symbolic representation to the meta RL setting in order to improve the generalization ability, reduce the computational cost, and increase the interpretability. Specifically, a gradient-based learning method is proposed to learn the contextual symbolic policy from scratch in an end-to-end differentiable way, which consists of a symbolic network, a path selector and a parameter generator.\n\n\nWeaknesses:\n- The literature review could include some previous work of combining symbolic representation with reinforcement learning in different ways, such as [1,2,3], although this work focuses on achieving the symbolic representation in a differentiable manner.\n- The abuse notation about the context of $c_{T}$: such as $c$ and $c_{\\kappa}$.\n- In Figure 1: it's not clear where do actor loss and critic loss come from. Is the \"context symbolic policy\" network only for the policy network? Does $Q(s,a,z)$ represent the value network? If this is the case, what's the architecture for the value network?\n- Are the symbolic operators listed in the paper applicable to all different tasks? If not, there should indicate what kinds of tasks can work with this symbolic operator library.\n- What's the experimental setup? Are those tasks trained alternatively or separately?\n- What are the differences for the paths in Table 2? Currently, I don't understand what are those.\n- The empirical evaluation seems weak and the current form of results is somewhat far from what was said to close the gaps in the introduction section, such as overfitting \\& poor generalization issue, inefficient deployment issues with limited computational resources, and explainability issue. For example, the problem size looks small. Even for the PEARL, the FLOPs are only in the level of KBytes and this might not be a big issue w.r.t. the deployment. For interpretability, it works well for the classical control problems since the original policy itself has a direct relationship between the state features and the action features. Although the proposed method can learn a symbolic policy, it looks to me that this is only an explicit expression for the learned policy which has partial interpretability rather than only numerical numbers from neural networks. The learned expression could vary if there are more different symbolic operators included.\n\n\nReferences:\n- [1] Garnelo, M., Arulkumaran, K., & Shanahan, M. (2016). Towards deep symbolic reinforcement learning. arXiv preprint arXiv:1609.05518.\n- [2] Garcez, A. D. A., Dutra, A. R. R., & Alonso, E. (2018). Towards symbolic reinforcement learning with common sense. arXiv preprint arXiv:1804.08597.\n- [3] Lyu, D., Yang, F., Liu, B., & Gustafson, S. (2019). SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 2970-2977).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject