Learning Intrinsic Symbolic Rewards in Reinforcement Learning
Authors: Hassam Sheikh
Shauharda Khadka
Santiago Miret
Somdeb Majumdar
Venue:  (2021)
Keywords: Reinforcement Learning
Intrinsic Rewards
Symbolic Regression

Abstract: Learning effective policies for sparse objectives is a key challenge in Deep Reinforcement Learning (RL). A common approach is to design task-related dense rewards to improve task learnability. While such rewards are easily interpreted, they rely on heuristics and domain expertise. Alternate approaches that train neural networks to discover dense surrogate rewards avoid heuristics, but are high-dimensional, black-box solutions offering little interpretability. In this paper, we present a method that discovers dense rewards in the form of low-dimensional symbolic trees - thus making them more tractable for analysis. The trees use simple functional operators to map an agent's observations to a scalar reward, which then supervises the policy gradient learning of a neural network policy. We test our method on continuous action spaces in Mujoco and discrete action spaces in Atari and Pygames environments. We show that the discovered dense rewards are an effective signal for an RL policy to solve the benchmark tasks. Notably, we significantly outperform a widely used, contemporary neural-network based reward-discovery algorithm in all environments considered.

Under review as a conference paper at ICLR 2021
LEARNING INTRINSIC SYMBOLIC REWARDS IN REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Learning effective policies for sparse objectives is key challenge in Deep Re- inforcement Learning (RL). common approach to design task-related dense rewards to improve task learnability. While such rewards are easily interpreted, they rely on heuristics and domain expertise. Alternate approaches that train neural networks discover dense surrogate rewards avoid heuristies, but are high- dimensional, black-box solutions offering little interpretability. present method that discovers dense rewards in the form of now-dimensional symbolic trees thus making them more tractable for analysis. The trees use simple functional operators to map agent's observations to scalar reward, which then supervises the policy gradient learning of a neural network policy. We test method on continuous action spaces in Mujoco and discrete action spaces in Atari and Pygame environments. We show that the discovered dense rewards are effective signal for an RL policy solve the benchmark tasks. Notably, we significantly outperform a widely used, contemporary neural-network based reward-discovery algorithm in all environments considered.
this paper, we
INTRODUCTION
RL algorithms aim learn a target task by maximizing the rewards pro-
Symbolic Tree
vided by the underlying environment. Only in few limited scenarios are the rewards provided by the environ- ment dense and continuously supplied to the learning agent, e.g. running score in Atari games (Mnih et al.) 2015), or the distance between the robot arm and the object in a picking task (Lillicrap et al., 2015). In many real world scenarios, these dense trinsic rewards are sparse or altogether absent.
Obs:
x1
X2
XN
Symbolic Reward
Generator
Obs
Action
Reward
Environment
Agent
Figure 1: LISR: functions and use it to train using standard Deep RL methods
agents discover latent rewards symbolic
In these environments, it is common approach to hand-engineer dense re- ward and combine with the sparse ob- jective construct surrogate reward. While the additional density leads faster convergence of a policy, creating surrogate reward fundamentally changes the underlying Markov Decision Process (MDP) formulation central many Deep RI. solutions. Thus, the learned policy may differ significantly from the optimal policy (Rajeswaran et al.) 2017, Ng et al. 1999). Morcover, the achieved task performance depends on the heuristics used to construct the dense reward, and the specific function used mix the sparse and dense rewards.
Recent works Pathha et al.) 2017 Zheng et al. 2018 Du et al. 2019 have also explored training a neural network to generate dense local rewards automatically from data. While, these approaches have sometimes outperformed Deep RL algorithms that rely on hand-de hand-designed dense rewards, they have only been tested in limited number of settings. Further, the resulting reward function estimators

---

Under review as a conference paper at ICLR 2021
black-box neural networks with several thousand parameters thus rendering them intractable to parse. A symbolic reward function lends itself better applications such as formal verification in AI and in ensuring fairness and removal of bias in the polices that are deployed.
In this paper, we present a method that discovers dense rewards in the form of low-dimensional symbolic trees rather than high-dimensional neural networks. The trees simple functional operators to map an agent's 's observations to scalar reward, which then supervises the policy gradient learning of a neural network policy. We refer to our proposed method Learned Intrinsic Symbolic Rewards (LISR). The high level concept of LISR is shown in Figure
To summarize, our contributions in this paper are:
We conceptualize intrinsic reward functions low-dimensional, learned symbolic trees constructed entirely of arithmetic and logical operators. This makes the discovered reward functions relatively easier to parse compared neural network based representations.
We deploy gradient- frree symbolic regression to discover reward functions. To the best of our knowledge, symbolic regression has previously been used to estimate optimal reward functions for deep RL.
RELATED WORK
The LISR architecture relies on the following key elements:
Symbolic Regression on a population symbolic trees to learn intrinsic rewards
Off-policy RL to train neural networks using the discovered rewards
Evolutionary algorithms (EA) a population of neural network policies search for optimal policy
Learning Intrinsic Rewards: Some prior works (Liu et al. 2014 Kulkarni et al. 2016 Dilok
thanakul et al., 2019 Zheng al. 2018) have used heuristically designed intrinsic rewards in RL settings leading to interesting lations such as surprise- based metrics (Huang et al. 2019). In this work, we benchmark against |Ptthak al. (2017) where Curiosity metric was successfully used to outperform A3C relatively complex environments like VizDoom and Super Mario Bros. LISR differs from these works that the reward functions discovered are low-dimensional symbolic trees instead of high-dimensional neural networks. Further, unlike LISR, that benchmark a single intrinsic reward approach on both discrete and continuous control tasks as well as single and multiagent settings.
not aware other works
Symbolic Regression in DL is well known search technique in the space of symbolic functions. A few works have applied symbolic regression to estimate activation functions (Sahoo et al.) 2018 8), value functions (Kubalík et al. 2019) and to directly learn interpretable RL policies in model based RL (Hein et al. 2018). To the best of our knowledge, symbolic regression has not previously been used to optimize for the reward function of an RL algorithm. For simplicity of design, we adopt classic implementation where a population of symbolic trees undergo mutation and crossover to generate new trees.
Evolutionary Algorithms (EAs) are class of gradient-free search algorithms Fogel 1995 Spears
et al. 1993) where population of possible solutions undergo mutate and crossover to discover novel solutions in every generation. Selection from this population involves ranking operation based on fitness function.
Recent works have successfully combined EA and Deep RI. accelerate learning. Evolved Policy Gradients (EPG) (Houthooft et al. 2018)) utilized EA to evolve a differentiable loss function parame- terized as convolutional neural network. CERL (Khadka et al. 2019) combined policy gradients (PG) and EA to find the champion policy based on fitness score. Our work takes motivation from both. Like EPG, we also search in the space loss functions albeit in the form of low-dimensional symbolic trees. Like CERL, we allow EA and PG leamers to share replay buffer to accelerate exploration. However, unlike LISR, CERL relies on access to an environment environment-rrovdded provided dense reward function for the PG learners.
2

---

Under review as a conference paper at ICLR 2021
3 LISR: LEARNING INTRINSIC SYMBOLIC REWARDS
The principal idea behind LISR to discover symbolic reward functions that then guide the learning of a policy using standard policy gradient methods. A general flow of the algorithm is shown in Figure 2
Two populations, comprising EA and SR learners respectively, are initialized. The EA population evolves using standard EA processes using a fitness function. In the SR population, each SR learner has corresponding symbolic tree that maps state observations tree represent simple mathematical or logical operators sampled from a pre-defined dictionary of operators or basis functions. The complete list of basis functions that are utilized by the symbolic trees is described in Appendix B The symbolic trees evolve using crossover and mutation based on fitness function - leading to the discovery novel reward functions. Figure depicts these operations on symbolic trees.
scalar reward. The nodes of the
Each SR learner uses reward to update its weights via policy gradient (PG) methods. We adopt Soft Actor-Critic (Haarnoja et al., (2017) for the continuous control tasks and Maxmin DQN {Lan) choice since they are both state-of-
cct al. 2020) for the discrete control tasks as the algorithms the-art methods in those respective environments. In either case, the reward used compute policy gradients is always an intrinsic, symbolic rewards and not any explicit dense reward provided by the environment.
The fitness function for any policy (SR EA) is computed as the undis- counted sum of rewards received from the environment, which is given only at the completion of an episode. Thus, any dense reward provided by the en- vironment is seen by any agent (SR or EA) only as a sparse, aggregated fitness function. At the end of each generation, all policies, EA and SR, combined, ranked and champion policy is selected.
Policies trained using symbolic rewards
Ranking and Champion Selection
Pollcy
Pollcy
Polley
SR Tree
SR Tree
3R Tree
Filness
Shared Replay Butfer: (s, s}
The shared replay buffer is the prin- cipal mechanism enabling sharing of information across the EA and the
SR learners in the population. Un- like, traditional EA where the data is discarded after calculating the fit- ness, LISR pools the experience for all learners (EA and SR) in the shared replay buffer identical to standard off-policy deep reinforcement learn- ing algorithms. All SR learners are then able to sample experiences from this collective buffer and use to gen- erate symbolic intrinsic rewards from their respective symbolic trees and update the policy parameters parameters using gradient descent. This mechanism maximizes the information extracted from each individual experiences.
Policy
Policy
Policy k
Policies trained using evolution
Figure 2: LISR: common replay buffer. A set of symbolic trees sample observations from this buffer and map them into scalar re- wards. SR learners also sample the same observations and the corresponding reward train using policy gradients. The champion policy (circled) is selected by ranking all policies, EA and SR, based on fitness function.
EA (bottom) and SR (top) learners share
This architecture is motivated by CERL (Khadka et al. 2019) where a common replay buffer between evolutionary and policy gradient learners was shown to significantly accelerate learning. In our experiments, we vary the proportion of EA and SR policies in order to distil the incremental importance of each to the final performance. For completeness, the LISR is shown Algorithm I
3

---

Under review as a conference paper at ICLR 2021
Mutation
Crossover
Gene
3R Tree
SR Tree
9R Tree
SR Tree 2
SR Tree
SR Tree 2
Figure 3: Evolution of symbolic trees. The colored polygons represent basic mathematical operators. For mutation (left), a random sub-tree replaced using another random sub-tree (gene). For crossover (right), parent trees swap sub-trees form child tree.
Algorithm LISR Algorithm
1: Initialize portfolio with SR learners - (Algorithm
2: Initialize population of l EA actors popa
3: Initialize an empty cyclic replay buffer R.
4: Define random number generator r() 10,
5: for generation = I, oc do
6: for actor pop, do
7: fitness, = Evaluate(T, R) + (Algorithm) 3 in Appendix)
8: Rank the population based fitness scores
9: Select the first e actors π E pop, as elites
10: Select (A-ce) actors æ from pop,, to form Set S using tournament selection with replacement
11:
while [S] < (k: - e) do
Use single-point crossover between random क E and S and append
for Actor Set do
14:
ifr() < &aut then Mutate(O") + (Algorithm in Appendix)
15:
16:
for Learner P do
Sample random minibatch of transitions (st, at, 8141) from R
18:
Compute reward f; LST (8;, 04, S1+1)
19:
Compute = f; + min T "00+,,,000000
j=1,2
20:
Update Lo by minimizing the loss: L; = + Ey: Lo, (8i, as (0)))22
21:
Update Lu using the sampled policy actions Soft update target networks:
22:
23:
Lig + T (1 Tip and
24:
L.grr c T.ge + (1 - T.ger
25:
for Learner L do
26:
score, R Evaluate R)
27:
Rank the learners P based on scores Select the first ý learners
28:
P as elites
29:
Select (m j)) symbolic trees ST from PSt, to form Set N using tournament selection. while [N] < (m - j) do Use iingle-oiitt crossover between random ST and ST N and append to N Use mutation between a random ST E and ST E and append to
30:
31:
32:
Algorithm 2 Symbolic Reward Learner
1:
procedure INITIALIZE Initialize actor ग and critic Q with weights #r and 82, respectively.
2:
3: Initialize target actor N' and critic Q' with weights ga and Øº' respectively.
4: Initialize the symbolic tree ST for reward generation
4

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
Our main objective is to demonstrate that LISR can be applied to problems involving continuous and discrete action spaces. To this end, evaluated LISR on Mujoco (Todorov et al.; 2012) for continuous control tasks and on Pygame (Qingfeng, 2019) and OpenAl-Gym Atari games (Brockman) al. 2016) for discrete control tasks. We evaluated LiSR's performance against three baselines: policies trained using standard EA implementation, policies trained using only intrinsic symbolic rewards and Curiosity where agents learn on combination of intrinsic rewards and environment- provided dense rewards.
For the continuous control tasks, we used Soft Actor-Critic (SAC) (Haarnoja et al. 2017) as our PG algorithm as it is the state-of wate-of-the-ar on a number of benchmarks. SAC is an off- pollppolicy actor-critic method based on the maximum entropy RL framework (Ziebart 2010). The goal of SAC to learn an optimal policy while behaving as randomly possible. This behavior encourages efficient exploration and robustness to noise and is achieved by maximizing the policy entropy and the reward.
For the discrete environments we adopt Maxmin DQN (Lan et al. 2020) which extends DQN (Mnih ct al. 2015) to addresses the overestimation bias problem in Q-learning by using an ensemble of neural networks to estimate unbiased Q-values.
Continuous control tasks: We evaluated on four environments from the Mujoco benchmark HalfCheetah, Ant, Hopper and Swimmer. We trained each environment with five random seeds for 150 million frames. We fixed the total population size (EA and SR learner) to 50 for all experiments. For LISR experiments,
kept the ratio between EA learners and SR learners equal 0.5. For the Curiosity experiments, we integrated the Intrinsic Curiosity Module (ICM) to work with SAC. We performed grid search for multiple learning rates for LISR, SR and Curiosity and report results corresponding to the best performing hyperparameters.
Curlosity
EA
1800
Average Return
SR
Average Return
4000
1600
1400
1200
2000
1000
1000
1.5
Frames (x10%)
Frames (x10%)
HalfCheetahh-v
Ant-v2
350
Average Return
1400
Average Return
300
1000
200
800
150
600
400
200
50
0
1.0
1.25 1.5
Frames (x109)
Frames (x109)
Hopper-v2
Swimmer-v2
Figure 4: Results on continuous control tasks in Mujoco. LISR outperforms all baselines except on the now-dimensional problem in Swimmer. Curiosity, with access to explicit as well as implicit rewards, is unable to learn an effective policy on any environment. SR on its own, with access only to intrinsic rewards, is also unable to scale but slightly outperforms Curiosity.
5

---

Under review as a conference paper at ICLR 2021
Our results the continuous control baselines are shown in Figure 4 We see that Curiosity fails to learn an effective policy on all environments even though it has access to the dense rewards in addition to its intrinsic rewards. SR on its own is also non-performant- however, it slightly outperforms Curiosity in 3 out environments. EA and LISR are both able to find effective policies. Notably, LISR outperforms EAA substantially in 3 out of 4 environments. On Swimmer, EAA slightly improves on sample efficiency although both EA and LISR find the optimal solution quickly. This finding is consistent with Khadka et al. (2019) that also showed that EA outperformed reinforcement learning on the relatively low dimensional problem in Swimmer. Since the key difference between EA and LISR the presence SR learners, these results demonstrate the incremental importance of the discovered symbolic rewards in solving the tasks.
Discrete control tasks: We evaluated LISR on four different discrete environments: Lunarl and Amidar, two high dimensional environments from Atari games and PixelCopter and Catcher, two low dimensional environments from Pygames. We trained multi-headed Maxmin DQN as policy gradient learner and used the Mean Vector regularizer (Sheikh & Bölöni 2020 to ensure diversity in the Q-values. Similar to the baselines in the continuous control experiments, we evaluated the performance of LISR against the performance of only EA, only SR learners and Curiosiry We trained PixelCopter, LunarLanda and Amidar for 50 million frames and Catcher for 30 million frames and show the results in Figure We observe that similar to the continuous control tasks, LISR outperforms all the baselines except Curiosity in the Catcher environment where the performance of both LISR and Curiosity are similar. Notably, in the PixelCopter and Catcher environments, the SR learners alone were able to achieve the maximum performance thus relying purely on discovered symbolic rewards. Curiosity significantly underperforms all baselines all except the Catcher environment. The complete list of hyperparameters is shown in Appendix C
80
Average Return
Average Return
60
Curiosity
EA
0.1
0.2
0.3
0.1
0.3
0.5
Frames (x10º)
Frames
(x10º)
Catcher
PixelCopter
160
200
Average Return
140
Average Return
100
0
100
-100
60
-200
-300
20
0.3 Frames ( (x108)
0.2
0.4
0.5
0.1
0.2
0.3
0.4
0.5
Frames (x108)
Amidar
LunarLander
Figure 5: Results discrete control tasks in Atari (top) and Pygames (bottom). LISR outperforms all baselines in all environments. Curiosity 's performance is overall significantly better on these tasks compared continuous control. SR on its own, with access environment provided dense rewards, is able to completely solve the Atari environments. SR on its own also outperforms Curiosity on all but the Catcher environment.
Multiplayer football: We also applied LISR to Google Research Football (Kurach et al.) 2020). physics-based, multiplayer 3D environment with discrete action spaces where multiagent teams aim

---

Under review as a conference paper at ICLR 2021
to score goals and maximize their margin of victory. The environment provides Scoring reward based on goals scored and a denser Checkpoint reward based on the distance of the ball to the goal.
Run to Score
Empty Goal
3 Keeper
Figure 6: Google Research Football environments used in the experiments
We test LISR on environment in the Foothall Academy set environments which describe specific game scenarios of varying difficulty. Specifically, we consider the following scenarios.
Run Score: Our player starts in the middle of the field with the ball, and needs score against an empty goal. Five opponent players chase ours from behind.
Empty Goal: Our player starts in the middle of the field with the ball, and needs to score against an empty goal.
3 vs with Keeper: Three of our players try to score from the edge of the box, one on each side, and the other at the center. Initially, the player at the center has the ball, and is facing the defender. There is an opponent keeper.
In the first two scenarios, we only control one player. the last scenario, we consider variations where control only one of our players and two of our players. In all cases, any player that do not control utilizes the strategy of a built-in Al bot. The scenarios are shown Figure 6
We benchmark LISR against EA and their published results with IMPALA (Espeholt et al. 2018 a popular distributed RI framework that was shown to outperform other popular algorithms like PPO (Schulman al.) 2017) and variations of DQN (Horgan al. 2018) this benchmark. For LISR, we only use the aggregated sum of rewards in an episode as a sparse fitness function. IMPALA, on the other hand, utilizes standard RL setup that exploits the dense rewards learn policy. Our goal was to investigate if LISR can be competitive with IMPALA with no access to the dense rewards.
5M (LISR)
50M (LISR)
5M (IMPALA)
(IMPALA)
(EVO)
50M (EVO)
1.00
Avg Goal Difference
0.80
0.40
0.20
0.00
Run to Score
Empty Goal
3 1 with Keeper
3 1 with Keeper
Figure 7: Experiments Google Research Football environments. Numbers in parentheses indicate number of players controlled by LISR
Figure 7 shows the performance on the four scenarios we evaluated. On the simpler environments involving empty goal, all three algorithms were able find performant solutions in less than 5M time steps. For the more difficult scenarios in involving players vs 1, IMPALA does outperform LISR. However, LISR is able to find competitive strategies compared IMPALA in both scenarios.
7

---

Under review as a conference paper at ICLR 2021
This is significant it shows that even in relatively complex. non-stationary multiagent scenar- ios, LISR is able to discover intrinsic symbolic rewards and be competitive with well-establiseed algorithms that exploit dense rewards.
key motivation to design LISR the discovery of symbolic reward func- tions that are involve many fewer operations than typical neural network based reward estimator. In all our experiments, we restricted the depth of the symbolic trees to operational layers in order to impose these constraints.
Discovered Rewards:
Consider the PixelCopter environment shown in Figure g It provides 8 state variables which are: 8o: position; 81: velocity; 82: distance to floor; 83: distance to ceiling; 84: next block's x distance to player; s5: next block's 's top y location and s6: next block's bottom y location and sy: agent's action.
Figure 9 shows an example of such a tree at the end of training. For better parsability, we unroll the tree into Python code. While we cannot claim that the particular reward function is interpretable, it is similar in structure to a classical symbolic rule and appears to rely on trigonometric transformations of positional variables. In this instance, it only utilizes 22 operations- thus making relatively easier to analyze compared to neural network reward estimators. In contrast, Curiosity 's ICM module that generates intrinsic reward is implemented as three neural networks with 5634 parameters.
Figure 8: Pixel- Copter environment
def get www.rrisss__ewwrdieeeeo
s_i, 2_2, s_3, s_4, s_5, e_s, s_7): P_1 (coo P_2 cas (s_3); p_3 pass_smaller P_2)
x_1 multiply(11, abs taubtract (s_2, P_31))
9_1 multiply aba (aubtract (1, (_41))
9.2
mex ([a_2, 2_7, e_,, 01)
9_3
max (2q_2, s_7, cos(0), multiplyis_0, s_6), miltipyy(__5, subtrast 1))1)
Y_1 div_bg_10(q_3)
y_2
square(s__]
y_3 cratacted_div(1, tiv__by_10(s__11
2_ gately_1, y_2, y_3)
equal_to X_1]
reward add;0, pes__mallerrdiv.byy.o.oe
z))
resard
Figure 9: An example a discovered symbolic reward on PixelCopter. We unroll the correspond- ing symbolic into Python-like code that be parsed and debugged. {s:} represent state observations.
CONCLUSION
framework that combines ideas from symbolic, rule-based machine learning with modern gradient- gradient-based learning. We showed that is possible to discover intrinsic rewards completely from observational data and train an RL policy. LISR outperformed other approaches that rely on neural network based reward estimators.
In this paper, we presented LISR
Our work is effort to bridge the interpretability gap in Deep RI.. While cannot claim that the discovered reward functions are interpretable, they are relatively easier to parse comprising of tens of symbolic operations compared thousands operations common in a neural network estimator. At the very least, this structure lends itself being more "human readable" compared to black box solutions. For example, shown in our example tree, LISR required only 22 operations compute reward in PixelCopter including simple trigonometric transforms on positional variables and one th-then-else gating condition. In scenario where a policy unstable, could be feasible trace the cause of instability to subset of those operations. This kind of "limited explainability" could be important for safety-critical applications like autonomous driving scenarios. Future work will focus on building the level of interpretability of the discovered functions.
One important drawback LISR the lack of sample-efficiency compared to established methods like SAC. This is somewhat expected as LISR operates with the key disadvantage of not having pre-defined dense reward signal. The primary bottleneck involves the search for an optimal reward function. In this work, implemented EA as the search mechanism. Future work will explore other alternatives like Monte-Carlo Tree Search (MCTS) as well as explore ways to turn off search when a reasonably good reward function has been discovered.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
N. Dilokthanakul, C. Kaplanis, N. Pawlowski, and M. Shanahan. Feature control as intrinsic motivation for hierarchical reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 30(11 ):3409-3410 2019.
Yali Du, Lei Han, Meng Fang, Ji Liu, Tianhong Dai, and Dacheng Tao. Lir: Learning individ- ual intrinsic reward in multi-agent reinforcement learning. In Advances in Neural Information Processing Systems 32, pp. 4403-4414, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pp. 1407-1416, 2018.
David Fogel. Evolutionary Computation: Toward a New Philosophy of Machine Intelligence. Wiley, 3rd edition, 1995.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Deep Reinforcement Learning Symposimm, 2017.
Daniel Hein, Steffen Udluft, and Thomas A Runkler. Interpretable policies for reinforcement learning by genetic programming. Engineering Applications of Artificial Intelligence, 76:158-169 2018.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In International Conference on Learning Representations, 2018.
forem/ 2 wot.ooormmmmiii-o--oo-aa--
Rein Houthooft, Richard Y. Chen, Phillip Isola, Bradly C. Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbecl. Evolved policy gradients, 2018.
Sandy H. Huang, Martina Zambelli, Jackie Kay, Murilo F. Martins, Yuval Tassa, Patrick M. Pilarski, and Raia Hadsell. Learning gentle object manipulation with curiosity-driven deep reinforcement learning, 2019.
Shauharda Khadka, Somdeb Majumdar, Tarek Nassar, Zach Dwiel, Evren Tumer, Santiago Miret, Yinyin Liu, and Kagan Tumer. Collaborative evolutionary reinforcement learning. International Conference Machine Leurning, pp. 3341-3350, 2019.
Jifí Kubalík, Jan Žegklitz, Erik Demer, and Robert Babuška. Symbolic regression methods for reinforcement learning. arXiv preprint arXiv: 2019.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Sacedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, pp. 3675-3683, 2016.
Karol Kurach, Anton Raichuk, Piotr Stanezyk, Michat Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly. Google research football: A novel reinforcement learning environment, 2020.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin Q-learning: Controlling the estimation bias of Q-lcarning. In Proceeding of the International Conference on Learning Represen- tations (2CLR- 2020. URL URL
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proc. of the 3rd Int't Conf. on Learning Representations (ICLR-2015), 2015.
B. Liu, S. Singh, R. L. Lewis, and S. Qin. Optimal rewards for cooperative agents. IEEE Transactions Autonomous Mental Development, 6(4):286-297, 2014.

---

Under review as a conference paper at ICLR 2021
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. RUsu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 February 2015. ISSN 00280836.
Andrew Y. Ng. Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, pp. 278-287, 1999.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Lan Qingfeng. Gym compatible games for reinforcenment learning. https://github.. (aan3/gym-gamam 2019.
Aravind Rajeswaran, Kendall Lowrey, Emanuel V. Todorov, and Sham M Kakade. Towards gen- eralization and simplicity in continuous control. In Advances in Neural Information Processing Systems 30, pp. 6550-6561, 2017.
Subham Sahoo, Christoph Lampert, and Georg Martius. Learning equations for extrapolation and control. In International Conference on Machine Learning, pp. 4442 4450, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 2017.
Hassam Ullah Sheikh and Ladislau Bölöni. Reducing overestimation bias by increasing representation dissimilarity in ensemble based deep q-learning. arXiv preprint arXiv:2006. 13823, 2020.
William Spears, Kenneth A De Jong, Thomas Bäck, David Fogel, and Hugo De Garis. An overview of evolutionary computation. In European Conference Machine Learning, pp. 442 459.. Springer, 1993.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 JEEE/RSJ International Conference on, pp. 5026- 5033, 2012.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances Neural Information Processing Systems, pp. 4644-4654, 2018.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. PhD thesis, Carnegie Mellon University, 2010.
10

---

Under review as a conference paper at ICLR 2021
APPENDIX
A ALGORITHMS
Algorithm 3 Function Evaluate
1: procedure EVALUATE(T, R)
2: fitness
3: Reset environment and get initial state So
4: while env is not done do
5:
Select action 0, (*) Execute action (4 and observe reward r2 and new state 81+-1 Append transition (st; der Tt: 8141) R
6:
7:
8:
fitness &- fitness + Tr and 8141
9: Return fitness, R
Algorithm 4 Function Mutate
1: procedure MUTATE((*))
2:
for Weight Matrix M E 0T do
3:
for iteration 1, frac M do
4:
Randomly sample indices i and.j from M's first and second axis, respectively
5:
if r()() supermutprol then
6:
M[i..j] = -Mi NO, 100 mut mutstrenghh)
7:
else if r() < reselprob then
8:
M Mii NO 1)
9:
else
10:
M Mii j] NOO mutsstrength)
B SYMBOLIC TREE DETAILS
The complete list of operators used for symbolic tree generation is shown below.
def add (left, right): return left + right
def subtract (left, right): return left right
def multiply (left, right): return lefhtright
def cos (left):
return up.cos (left)
def sin (left):
return sinsin (left)
def tan (left):
return tan (left)
def max (nums):
return np. maxmimum (nums)
def min (nums):
return np. mi n mum (nums)
11

---

Under review as a conference paper at ICLR 2021
def lass_greaterrre (left, right): :
if left right: return left return right
def pass_ smaller right): if left right: return left return right
def equal_to (left, right): return float (left right)
def gate (left, right, condtion); if condtion <= 0:
return left
else:
return right
def square (left); return left+left
def is is__egaaive (left))
if left < 0: return 1.0 return 0.0
def div_by_100 :left):
return 1eft/100.0
def 10 return left/10.0
def protected_div(left, right):
with np.errstate (divide='ignore) invalld='ignore )
x di.iivide (left, right)
if isinstance (x, np.rdarray):
x [np..sinf(() =
[np. isnan (x)]
elif ipiisinf (x) np.isnan (x)
x
return
12

---

Under review as a conference paper at ICLR 2021
C IMPLEMENTATION DETAILS
The complete list of hyperparameters used for LISR experiments are given below. For football experiments, used the same hyperparameters that we used discrete control tasks.
Table I: Hyperparameters for LISR for continuous control tasks
Hyperparameter
Value
Population Size
Target Weight
le-3
Actor Learning Rate
[le 3 le 4
3e 5
Critic Learning Rate
[le 3 le-d 5
Replay Buffer
1e6
Batch Size
[256, 1024]
Exploration Steps
5000
Optimizer
Adam
Hidden Layer Size
256
Mutation Probability mutppro
0.9
Mutation Fraction muac
0.1
Mutation Strength muustrength
0.1
Super Mutation Probability supermutprob
0.05
Reset Mutation Probability resetmutprod
0.05
Number of elites e
7%
Table 2: Hyperparameters for LISR for discrete control tasks
Hyperparameter
Value 50
Population Size k
Target Weight
le
Actor Learning Rate
[1e 33 le-4)
Maxmin DQN Heads
Regularization Weight
le
Replay Buffer
1
Batch Size
[64, 256]
Exploration Steps
5000
Optimizer
Adam
Hidden Layer Size
256
Mutation Probability mulprob
0.9
Mutation Fraction mullfrac frac
0.1
Mutation Strength mutstrength
0.1
Super Mutation Probability supermut
0.05
Reset Mutation Probability reselmulproo.
0.05
Number of elites
7%
13


Reviews:
[{"review_id": "w4ZSa9h-rQe", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes LISR, a method for learning symbolic intrinsic reward functions from interaction using using symbolic regression via \"symbolic trees\". The approach is tested on a few continuous control Mujoco tasks, as well as a few discrete control gym environments and a few benchmarks based on the Google Research Football (GRF) environment.\n\nGood things:\n\n- The idea of using symbolic regression to discover intrinsic reward functions is extremely interesting, and a potentially good usage of symbolic regression towards enabling better interpretability of complex RL learning pipelines.\n- Testing on multiple environments with wildly different tasks, control systems, and agent settings is excellent, and shows that the method is potentially applicable to a wide variety of settings.\n\nConcerns:\n\n1. The explanation of the method is somewhat poor, and almost entirely left to the (nonetheless good!) algorithm. The overall system integrates a lot of moving parts: a RL policies, multiple disjoint populations, a simple symbolic regression model, a shared experience replay buffer, etc. -- However, the manuscript is lacking of rationalisation about choices made when assembling this system is lacking (and effectively only the intuition behind the shared replay buffer is properly provided to the reader). This raises multiple questions, such as:\n  a. Why were two different types of populations used?\n  b. Why is the fitness evaluation mechanism shared among these two sets?\n  c. Why was the fitness system used to rank policies, rather than reward models?\n  d. Why was all of this not achievable simply through backpropagation?\n\nMy guess is that much of these details are contained in the CERL paper, however considering that the proposed pipeline largely seems to resemble it, it would probably be best to provide an overview of it.\n\n2. It seems a little unfair to claim that LISR was tested on \"multiagent\" scenarios. GRF was mostly used as a single-agent environment, and the manuscript lacks in details about the setting with two agents. It would be good to add details about how the 2-agent setting was constructed (e.g. is there a centralised policy? Is there a joint action space? Is the reward function decomposed?), to understand exactly how LISR is operating in this setting.\n\n3. Considering the significant amount of moving parts in the method, experimental evaluation section could use more ablations of the system and some more analysis. It would be good to understand for instance:\n   a. how the system behaves with varying sizes of the populations;\n   b. whether the reward trees look similar between multiple training runs / seeds;\n   c. How the learnt trees differ between each environment setup...\n... and so on.\nThis is particularly important for the narrative of the paper, since the method aims to improve interpretability of reward functions in RL agents for real-life tasks. It could also provide data to come up with suggestions on how to improve LISR to bridge the gap against other SOTA algorithms.\n\n\nTo conclude,  I currently cannot recommend acceptance, however I'd be willing to revise my score provided that at least some of the following improvements are made:\n- Better description of the method, its assumptions, and how it specifically builds on / differs from CERL;\n- More details about the output of the trained reward systems, and some analysis about why they produce what they produce (beyond empirical testing on multiple envs);\n- More details about the multi-agent setup.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ip9jPKFvUiS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of review:\nInteresting work, but insufficient support of the main claim on interpretability.\n\nDescription:\nThis paper proposes a mechanism to learn reward functions that is based on a symbolic reward generator (in the form of a tree of pre-defined simple math operators).  The generated reward is used to optimize a policy, and an evolutionary approach is used over a set of these policies to select the final “champion” policy.  Observed rewards from the environment are used to rank the policies (rather than to directly optimize the policy).\n\nThe main claim of the paper is that this method is more interpretable, since presumable the learned reward function, in the form of this tree of simple operators, can be more easily interpreted than a general neural network (or other complex non-linear) approximator.\n\nStrengths:\n-\tPaper is well written, easy to understand.  Good use of images to illustrate the method.\n-\tThe proposed method is based on recent literature, and seems algorithmically sound.\n-\tThere are several empirical results to characterize the performance, on diverse benchmarks and different baselines.\n-\tThe authors are fair in characterizing their results.  They do not overclaim interpretability of the results presented (Fig.9, Sec. 4 & 5).\n\nWeaknesses:\n-\tThe motivation, of extracting interpretable rewards, is an interesting one.  Though the authors do not make a very convincing case for when this is absolutely necessary. The reality is that for most real-world tasks in RL (and I have worked on many!), the reward is specified by a human, and thus is interpretable from the start.  I can imagine some cases in imitation learning or inverse RL, where one has expert trajectories (e.g. driving a car in dense traffic) where it might be useful to infer an interpretable reward, but this is not the setting here.\n-\tThe paper does not describe the formal setting.  It took me a while to figure out if/how observed rewards were used.\n-\tThe authors do not provide a (time) complexity analysis of their algorithm, compared to others.\n-\tSome of the plots (Fig.4, 5) are hard to read, colours are too similar.  Fig.7 does not include error bars.\n-\tThe empirical results on the symbolic reward being interpretable are not supported.  I tried to read the example in Fig.9, and it makes no sense.  Quantities with very different units are shoved into a function together, etc. \n-\tThere are too many results on performance (which is not a main claim) vs interpretability (which is the main motivation).\n\nQuestions:\n-\tWhile I am not convinced by the interpretability of the method, the empirical results are still quite good. In general, I would expect to pay a performance cost for a more interpretable solution (that is usually the case in many other works on interpretable ML, due to using a simpler function.  Is it possible that here, the good performance is due to the symbolic reward estimator acting as a regularization mechanism?  Do you have any other explanation why the results are good?   Can you discuss in more detail the trade-offs between interpretability & performance?\n-\tI am not sure I understand exactly what each of the alternative methods considered in the empirical results are doing. Do you have a case that just uses the observed reward directly for policy gradient?   Which case would this be?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GZboqjM96Nw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to learn symbolic regression (tree) to make analysis of reward function more interpretable and tractable. Authors combine the benefits of interpretable SR learner and diversity driven evolutionary algorithms. Authors conduct the training of policy in the shared replay buffer for off-policy RL.\n\nStrong points\n- An interpretable way of policy learning using symbolic reward functions\n- New interpretable reward as shown in Figure 9.\n\nWeak points\n- Symbolic reward functions are not clearly written (or not self-contained)\n- It is hard to find algorithmic novelty in the paper\n- Empirical evaluations are not extensive. \n- It would be better to include examples of new discovery of interpretable reward function\n\nThe ideas in the presented paper looks reasonable. However, the symbolic reward function and the procedure to learn such symbolic representation is not clearly written. Also, it is hard to see that the proposed algorithm present new qualitative (interpretable) discovery of reward function. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an algorithm to learn symbolic intrinsic rewards via a symbolic function generator. The policy optimizes this reward function and an evolutionary algorithm selects between a set of such policies. The core idea is that learning with such a symbolic reward function is useful in sparse reward environments and also enables better interpretability.

${\bf Pros}$:
1. The learnt reward function has a relatively simple form and is therefore interpretable
2. The experimental section is quite extensive ranging from diverse tasks, control systems and agent systems. However there are some issues about showing clear need of the proposed method

${\bf Cons}$:
1. There was a consensus among reviewers that the paper does not make a strong case for the symbolic reward generator. In the rebuttal the authors argued that as RL scales to real world problems, it will become necessary to use such a method. I can understand how it would be useful in the context of inverse RL or imitation learning. However, as R3 points out, in the cases considered in this paper, the rewards are fairly intuitive and explainable. The paper might become stronger by directly tackling problems with such constraints.
2. There is confusion about the details and scope in the current version of the paper. The paper would become stronger by incorporating all the feedback received during the review period. 
Decision: Reject