Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a method to learn symbolic regression (tree) to make analysis of reward function more interpretable and tractable. Authors combine the benefits of interpretable SR learner and diversity driven evolutionary algorithms. Authors conduct the training of policy in the shared replay buffer for off-policy RL.

Strong points
- An interpretable way of policy learning using symbolic reward functions
- New interpretable reward as shown in Figure 9.

Weak points
- Symbolic reward functions are not clearly written (or not self-contained)
- It is hard to find algorithmic novelty in the paper
- Empirical evaluations are not extensive. 
- It would be better to include examples of new discovery of interpretable reward function

The ideas in the presented paper looks reasonable. However, the symbolic reward function and the procedure to learn such symbolic representation is not clearly written. Also, it is hard to see that the proposed algorithm present new qualitative (interpretable) discovery of reward function. 
review_title: It is not clear which models are used in symbolic regression

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
The paper proposes LISR, a method for learning symbolic intrinsic reward functions from interaction using using symbolic regression via "symbolic trees". The approach is tested on a few continuous control Mujoco tasks, as well as a few discrete control gym environments and a few benchmarks based on the Google Research Football (GRF) environment.

Good things:

- The idea of using symbolic regression to discover intrinsic reward functions is extremely interesting, and a potentially good usage of symbolic regression towards enabling better interpretability of complex RL learning pipelines.
- Testing on multiple environments with wildly different tasks, control systems, and agent settings is excellent, and shows that the method is potentially applicable to a wide variety of settings.

Concerns:

1. The explanation of the method is somewhat poor, and almost entirely left to the (nonetheless good!) algorithm. The overall system integrates a lot of moving parts: a RL policies, multiple disjoint populations, a simple symbolic regression model, a shared experience replay buffer, etc. -- However, the manuscript is lacking of rationalisation about choices made when assembling this system is lacking (and effectively only the intuition behind the shared replay buffer is properly provided to the reader). This raises multiple questions, such as:
  a. Why were two different types of populations used?
  b. Why is the fitness evaluation mechanism shared among these two sets?
  c. Why was the fitness system used to rank policies, rather than reward models?
  d. Why was all of this not achievable simply through backpropagation?

My guess is that much of these details are contained in the CERL paper, however considering that the proposed pipeline largely seems to resemble it, it would probably be best to provide an overview of it.

2. It seems a little unfair to claim that LISR was tested on "multiagent" scenarios. GRF was mostly used as a single-agent environment, and the manuscript lacks in details about the setting with two agents. It would be good to add details about how the 2-agent setting was constructed (e.g. is there a centralised policy? Is there a joint action space? Is the reward function decomposed?), to understand exactly how LISR is operating in this setting.

3. Considering the significant amount of moving parts in the method, experimental evaluation section could use more ablations of the system and some more analysis. It would be good to understand for instance:
   a. how the system behaves with varying sizes of the populations;
   b. whether the reward trees look similar between multiple training runs / seeds;
   c. How the learnt trees differ between each environment setup...
... and so on.
This is particularly important for the narrative of the paper, since the method aims to improve interpretability of reward functions in RL agents for real-life tasks. It could also provide data to come up with suggestions on how to improve LISR to bridge the gap against other SOTA algorithms.


To conclude,  I currently cannot recommend acceptance, however I'd be willing to revise my score provided that at least some of the following improvements are made:
- Better description of the method, its assumptions, and how it specifically builds on / differs from CERL;
- More details about the output of the trained reward systems, and some analysis about why they produce what they produce (beyond empirical testing on multiple envs);
- More details about the multi-agent setup.

review_title: More details and analysis about the method are needed

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary of review:
Interesting work, but insufficient support of the main claim on interpretability.

Description:
This paper proposes a mechanism to learn reward functions that is based on a symbolic reward generator (in the form of a tree of pre-defined simple math operators).  The generated reward is used to optimize a policy, and an evolutionary approach is used over a set of these policies to select the final “champion” policy.  Observed rewards from the environment are used to rank the policies (rather than to directly optimize the policy).

The main claim of the paper is that this method is more interpretable, since presumable the learned reward function, in the form of this tree of simple operators, can be more easily interpreted than a general neural network (or other complex non-linear) approximator.

Strengths:
-	Paper is well written, easy to understand.  Good use of images to illustrate the method.
-	The proposed method is based on recent literature, and seems algorithmically sound.
-	There are several empirical results to characterize the performance, on diverse benchmarks and different baselines.
-	The authors are fair in characterizing their results.  They do not overclaim interpretability of the results presented (Fig.9, Sec. 4 & 5).

Weaknesses:
-	The motivation, of extracting interpretable rewards, is an interesting one.  Though the authors do not make a very convincing case for when this is absolutely necessary. The reality is that for most real-world tasks in RL (and I have worked on many!), the reward is specified by a human, and thus is interpretable from the start.  I can imagine some cases in imitation learning or inverse RL, where one has expert trajectories (e.g. driving a car in dense traffic) where it might be useful to infer an interpretable reward, but this is not the setting here.
-	The paper does not describe the formal setting.  It took me a while to figure out if/how observed rewards were used.
-	The authors do not provide a (time) complexity analysis of their algorithm, compared to others.
-	Some of the plots (Fig.4, 5) are hard to read, colours are too similar.  Fig.7 does not include error bars.
-	The empirical results on the symbolic reward being interpretable are not supported.  I tried to read the example in Fig.9, and it makes no sense.  Quantities with very different units are shoved into a function together, etc. 
-	There are too many results on performance (which is not a main claim) vs interpretability (which is the main motivation).

Questions:
-	While I am not convinced by the interpretability of the method, the empirical results are still quite good. In general, I would expect to pay a performance cost for a more interpretable solution (that is usually the case in many other works on interpretable ML, due to using a simpler function.  Is it possible that here, the good performance is due to the symbolic reward estimator acting as a regularization mechanism?  Do you have any other explanation why the results are good?   Can you discuss in more detail the trade-offs between interpretability & performance?
-	I am not sure I understand exactly what each of the alternative methods considered in the empirical results are doing. Do you have a case that just uses the observed reward directly for policy gradient?   Which case would this be?

review_title: Learning Intrinsic Symbolic Rewards in Reinforcement Learning

Meta-review: This paper proposes an algorithm to learn symbolic intrinsic rewards via a symbolic function generator. The policy optimizes this reward function and an evolutionary algorithm selects between a set of such policies. The core idea is that learning with such a symbolic reward function is useful in sparse reward environments and also enables better interpretability.

${\bf Pros}$:
1. The learnt reward function has a relatively simple form and is therefore interpretable
2. The experimental section is quite extensive ranging from diverse tasks, control systems and agent systems. However there are some issues about showing clear need of the proposed method

${\bf Cons}$:
1. There was a consensus among reviewers that the paper does not make a strong case for the symbolic reward generator. In the rebuttal the authors argued that as RL scales to real world problems, it will become necessary to use such a method. I can understand how it would be useful in the context of inverse RL or imitation learning. However, as R3 points out, in the cases considered in this paper, the rewards are fairly intuitive and explainable. The paper might become stronger by directly tackling problems with such constraints.
2. There is confusion about the details and scope in the current version of the paper. The paper would become stronger by incorporating all the feedback received during the review period. 
Decision: Reject