Reasoning With Hierarchical Symbols: Reclaiming Symbolic Policies For Visual Reinforcement Learning
Authors: Wenqing Zheng
S P Sharan
Zhiwen Fan
Zhangyang Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: Symbolic Regression
Reinforcement Learning
Convolutional Neural Networks
Interpretability

Abstract: Deep vision models are nowadays widely integrated into visual reinforcement learning (RL) to parameterize the policy networks. However, the learned policies are overparameterized black boxes that lack interpretability, and are usually brittle under input distribution shifts. This work revisits this end-to-end learning pipeline, and proposes an alternative stage-wise approach that features hierarchical reasoning. Specifically, our approach progressively converts a policy network into the interpretable symbolic policy, composed from geometric and numerical symbols and operators. A policy regression algorithm called RoundTourMix is proposed to distill the symbolic rules as teacher-student. The symbolic policy can be treated as discrete and abstracted representations of the policy network, but are found to be more interpretable, robust and transferable. The proposed symbolic distillation approach is experimentally demonstrated to maintain the performance and ``de-noise" the CNN policy: on six specific environments, our distilled symbolic policy achieved compelling or even higher scores than the CNN based RL agents. Our codes will be fully released upon acceptance.


Under review as a conference paper at ICLR 2022
REASONING WITH HIERARCHICAL SYMBOLS: RE- CLAIMING SYMBOLIC POLICIES FOR VISUAL REIN- FORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep vision models nowadays widely integrated into visual reinforcement learning (RL) to parameterize the policy networks. However, the learned policies are overparameterized black boxes that lack interpretability, and usually brittle under input distribution shifts. This work revisits this end-to-end learning pipeline, and proposes an alternative stage-wise tagee-wiseaae approach that features hierarchical reason- ing. Specifically, our approach progressively converts policy network into the interpretable symbolic policy, composed from geometric and numerical symbols and operators. A policy regression algorithm called RoundTourMix is proposed to distill the symbolic rules as teacher-student. The symbolic policy can treated discrete and abstracted representations of the policy network, but are found more interpretable, robust and transferable. The proposed symbolic distillation approach is experimentally demonstrated to maintain the performance and "de- noise" the CNN policy: on six specific environments, our distilled symbolic policy achieved compelling
higher scores than the CNN based RL agents. Our codes will be fully released upon acceptance.
detilatoon kara draw aucllary locc suaste
propoed disillatin
In Ingger octicn
Viletts could be incaared
propercd
90
Figure 1: Four example environments adopted for distilling the CNN policy network knowledge into the symbolic policy. From left to right are: Airstriker-Genesis. Pong, CircusCharlie, Seaquest.
INTRODUCTION
Reinforcement learning (RL) is effective to explore complex environments through interactions (Kaiser et al., 2019). central target in building stronger RL algorithms to learn reliable and generalizable policies, robust unseen real-world data with possible shifts. This is difficult task, especially when the input space is high-dimensional or continuous, e.g., visual data, and the sampled observations during training policies from those data, it is common to parameterize policies as numerical predictors, e.g, neural networks (Francois-Lavet al. 2018). Despite their success, the neural network based RL. algorithms require prohibitive amounts of data to train, and the resultant rules are "black boxes" that are hardly interpretable (Mnih al. 2013 Heuillet al. (022]) nor stable under input domain shifts (Schmeckpeper et al. 2020
hence destined to be extremely sparse. To learn sophisticated
To mitigate this gap, let us take inspirations from ourselves. Humans integrate information at multiple levels of abstraction: from sensory, concept symbols, and to underlying

---

Under review as a conference paper at ICLR 2022
"computational" mechanisms (Timpf et al. 1992). The hierarchical and progressive integration of concept abstraction serves as the key for intermediate states to enable interpretable logical reasoning as well as for generalization: the latter because appropriate abstraction can eliminate nuisances (Krawczyk et al.] (01T). Therefore, we are motivated to program our visual RI. policies into an alternative representation of symbolic rules. Compared to its numerically parameterized counterparts, a symbolic rule is significantly lighter-weight, likely more interpretable, and potentially more generalizable.
Abstracting pure symbolic rules from data not new The family of symbolic regression (SR) methods (Cranmer et al. 2020 Runarsson & Jonsson 2000 Gustafson et al.) 2005; (Orchrdd & Wang 2016) have been studied to directly search from the discrete symbolie space to compose target equations.
Different from conventional regression techniques that optimize pre-specified model structure, SR infers both model structures and parameters from data. Most popular SR algorithms rely on genetic programming to evolve symbolic math equations from scratch (Runarsson & Jonsson 2000) Gustafson et al.] 2005 Orchard & Wang 2016). However, compared to the differentiable learning of numerical policy networks, those SR methods are slow on large/complicated problems and rely on many heuristies to work (Runarsson & Jonsson 2000; Gustafson et al.] 2005 Orchard & Wang
2016). Hence, naively plugging those methods as policy learners for visual RL, c.g., to initialize a population of simple symbolic policies and then gradually mutate/select better candidates while
interacting with the environment, will be computationally prohibitive (Runarsson & Jonsson| 2000 Gustafson et al. 2005; Orchard & Wang 2016 in addition to suffering from the sparse rewards.
We thus propose to synergize the numerical and symbolic policy learning, hoping to gain the best from both worlds. Inspired by the idea of "knowledge distillation" (Hinton et al., 2015),, propose a symbolic policy distillation approach: it does not directly compose and evolve the symbolic policy through RL interaction, but instead, distills a symbolic policy from the behavior of a numerical policy network as the "teacher". After the teacher is learned via conventional RL., our approach extracts symbolic form that approximates the relationship between the input visual observations and the output actions of the teacher. A key barrier here is, however, the fact that the teacher's 's input is an image, while symbolic policy cannot easily consume high-dimension continuous input. robust parsing mechanism is thus needed to abstract an image into a set of symbolic input operands, on top of which more symbolic operators can be defined to compose them. That requires us cross the famous "semantic gap" (Hein) 2010) in vision, by properly discretizing the perception-decision procedure.
In view of the above challenges, we study the progressive discrete symbolization of representations, from the continuous visual observations to the discrete actions, and accordingly design a symbolic policy distillation procedure that leverages the resultant hierarchical symbols. The progressive symbolization for visual RI. bridges the raw image input, the desired symbolic input operands, and the output action space. To achieve so, we first use an object detection module to achieve the "first level" of discretization, i.e., from the input image to discrete set of objects (both classes and locations). We then proceed to the "second level", to (iteratively) search the symbolic rule that maps the input set of objects to the output set of actions. The second level is accomplished using symbolic teacher-studed hierarchical, explainable, and generalizable symbolic rule in the form of symbolic compositions, that can be treated as discrete and abstracted representations of the visual policy network.
distillation algorithm named RoundTourrMix. Our entire process yields a
As observed through extensive experiments, our symbolic policy distillation approach captures the underlying structure in white-box form through data-driven experience, and meanwhile gets rid of visual domain nuisances through state abstraction (e.g., detection). In particular, the distilled symbolic policy leverages the representation of object location, velocity and class information, which naturally group two scenarios (visual scenes) together if the same logical specifications are congruent even if the scenes' pixel-level attributes are markedly different. The distillation procedure also requires orders of magnitude less data than policy neural network training. Our approach paves new way to make the visual RL more interpretable, reliable and generalizable, and also narrows the gap between numerical and symbolic approaches in RL. Our concrete technical contributions are summarized as follows:
We propose novel progressive discretization of the perception-decision procedure with three logical hierarchies: from a raw image to its object-level abstraction (called "geometric symbols" hereinafter), to the numerical state and discrete action spaces.
2

---

Under review as a conference paper at ICLR 2022
We propose novel policy regression algorithm that distills the teacher behaviors of policy networks into symbolic rules. Our approach termed RoundTourMix is gradient free and data-efficient to train, and the output rule enjoys good interpretability and reliability.
Through perfommaaccnneeppptolliyttttatee cies demonstrate comparable or even stronger effectiveness in seven highly challenging visual RL environments. They achieve the comparative higher scores with interpretable actions while generalizing better on the new environments than their teacher policy networks.
evaluation, distilled symbolic poli-
RELATED WORKS
Our work tightly intersects with interpretable reinforcement learning, and mathematical symbolic regression. In reinforcement learning, the way to boost interpretability can be roughly categorized into (a) intrinsic interpretability based methods and (b) post-hoc explanation based methods. The Intrinsic interpretability based methods require the learned models be self-nndertta by learning over interpretable architectures. Among the existing methods, the authors Lyu et al. (2019) proposed learn an interpretable policy online, Zambaldi al. (2018); Jiang & Luo (2019); Dong et al. (219)) to introduce architectural inductive biases to learn a interpretable policy, Dittadi et al. (2020) to perform width-based planning on relevant features extracted from game frames, and Ma et al. ) preserves interpretability by learning set of logical rules. Other methods also leverage imitation learning (Verma et al. 2018 2019: Bastani et al.] 2018). For post-hoc explanation based methods, the papers in Zahavy et al. (2016); Greydanus et al. (2018); Gupta et al. (2019) propose to use t-SNE and saliency map to explain the learned policy. (Liang et 2015) proposes to capture key features of DQN into practical linear representations. Other methods include attention based method (Shi et al., 2020), visual summary based method (Sequeira & Gervasio (020), reward decomposition based method (Juozapaitis et al. 2019), casual model based method (Madumal et al.. 2020), Markov chain based method (Topin & Veloso 2019) and case-based expert-behaviour retrieval method Ontanón et al., 2007).
Our symbolic distillation falls within the post-hoe methods, as we first learn CNN policy that enjoys the neural network's 's free form optimization, then distills it into an explainable symbolic policy. Connecting existing methods that follow this path, the work in (Coppens et al.) (2019) distill the learned policy into a soft decision tree. Their visual observations are coarsely quantized into 10 x 10 cells, and the soft decision tree policy is learned over the 100 dimensional space. Garnelo al. (2016): Garcez et al. (2018) also learn symbolic RL algorithms, but the states of the environments they considered are clean and simple, and are not as challenging as video games.
Symbolic Regression (SR) Runarsson Jonsson 2000 Orchard & Waan 2016 Schmidt & Lipson
2009 Cranmer et al. 2020 Cranmer 2020 Petersen el al. 2019; Gustafson et al. 2005) is a recently emerging and promising approach for discovering underlying symbolic rules of the observed data. In discovering the mathematical equations, several algorithms adopted the gradient- free genetic programming method. These algorithms generate populations of candidates equations, mutate them and select better candidates, then repeat this evolution procedure until certain thresholds of maximum interation number or performance level are met. Examples include Eureqa (Schmidt & Lipson 2009)
and PySR (Cranmer et al.] 22200] Cranmer (020), and few others (Runarsson & Jonsson 2000
Orchard & Wang] 2016]] Gustafson et al. 2005). Some other methods leverage the deep neural network to parse out equation trees, such as (Petersen et al. 2019 Bello et al. 2017).
3 THREE TYPES OF IMAGE DATA REPRESENTATIONS
In this work, we focus on the visual Reinforcement Learning (RL) domain, and distill the teacher agent behavior into a symbolic form. We use the Gym Retro games (Nichol et al.) as primary examples to study policy distillation. For the most common visual RL case, the agent is modeled by an end-to-end neural network (usually based on a convolutional neural network, or CNN) that maps the image type data to the action. As pixel level data representation, the raw image offers rich low level information but without abstraction. On the other hand, as a high level decision regime, the symbolic rule takes robust abstracted symbols as its input operands.
3

---

Under review as a conference paper at ICLR 2022
Visual Observation
Geometric Symbol Representation
Numerical State
0.25
-0.49
11.30
Action
Shortcut on Atari achieve
Draw auxiliary lines, measure the geometric relations to:
Object Detection: Optical Flow estimation 2. Shape based object classification
1. Construct Numerical State Provide input operands for distillation
Figure 2: The discretized perception-decision procedure, with three representation types (see Sec. 3).
To bridge this gap, hence necessary summarize the image pixels into symbolic operands (also refered to geometric objects in this work) as required by the symbolic policy. Besides the object level abstraction the raw image, richer numerical form representations are also needed, as the numbers are the basic building blocks to effectively represent knowledge and conduct learning. Summarizing these motivations, we introduce three types representations, the visual observation, the geometric symbol representation, and the numerical state.
The initially obtainable representation is the visual observation, which contains the raw information and serves as the input for the CNN based model. With these visual observations, we run a (pre- trained and fixed) object detection algorithm to obtain object lists in the images. The object detection module will provide us the class, location, as well as the velocity of every single detected object in the image sequence. The output of the object detection module is the second type of image data representations, which we term as the geometric symbol representation. The geometric symbol representation the desired symbol abstraction of the pixel level image, and supplies the input operands for the symbolic distillation to work on. We use optical flow to segment and estimate the velocity of the objects, and further classify the objects based on their shapes. More details are in Appendix B Eventually, the third type of representation is the numerical state, which is obtained by drawing auxiliary lines, and prasuringgquantiiing certain relationships of the geometric symbols (distances, orientations, etc). The numerical state is then used to predict the final action.
Overall as illustrated in Fig. in order to parse a raw image into a symbolic policy, the perception-decision procedure discretized into the following steps: visual observation + geometric symbol representati on -> numeri cal state action.
LEARNING To DISTILL TEACHER BEHAVIOR INTO SYMBOLIC POLICY
With the discretized decision procedure defined above, the remaining question is how to concretely distill the teacher behavior into the symbolic policy. In what follows, we summarize the learning/dis- tillation objectives in section 4.1 then describe the components of symbolic search space in section and finally describe the distillation algorithm in section
4.1 THE LEARNING OBJECTIVES
Over the three representations discussed above, hierarchical symbolic reasoning proce- dure can be built. geometric symbol representation! could be robustly implemented and is reliable for the usage of future steps, therefore, we set as learning-less. Based on the observed geometric symbol representation, we learn the rest of reasoning steps [geometris symbol representation + numerical state + action] through the proposed symbolic distillation.
this work, we posit that the step [visual observation -
Intuitively, there are three reasoning components subject to learning: what geometric relation- ship to measure out of the geometric symbol combinations, as compose the numerical state; @ what is the condition to take action based on the measured numerical state; and how to take action, under the current condition. In this research, the learning targets and O are usu- ally found to be stacked with each other in a hierarchical way, yielding a tree-shaped decision procedure (Sutton et al. 1999 Dietterich] 2000 Botvinick et al. 2009). To connect these learning
4

---

Under review as a conference paper at ICLR 2022
targets with the discretized decision procedures in section the [geometric symbols - numerica memerrical state] step, and and @ determine the learning target for the [numerical state - action] step. In the following section, we define and construct the basic building block to compose these learning targets.
3 determines the learning target for
4.2 SYMBOLIC POLICY SEARCH SPACE
To make symbolic policy distillation executable, have to answer two open questions: what are the available building blocks to construct symbolic policy, and how can we construct it? In the following, we answer them by defining the symbolic search space, which contains the geometric operator search space and the numerical operator search space.
The geometric operator search space. The geometric operator search space is designed for the learning target 1 in section 4.1 and the components within this space are geometric operator com- binations applied to detected objects. To enable stronger expressiveness, we include two types of geometric operators: the auxiliary line drawer and the geometric attribute measurer. The auxil- iary line drawer takes the input of detected objects, and draw auxiliary lines according to certain given descriptions. This type of operators include ve woo.ity.eetteniineooooooooooo..e (extend the line along certain object's 's velocity direction, and keep track of it), vortical/horizonta//ill auxiliary lines), and intersection marker (mark the intersection be- tween the auxiliary lines and/or detected objects, and keep track of it). The newly drawn auxiliary lines/intersection points are added into the collection of the geometrie symbol representation.
(drawing static
For the second type of geometric operators, the geometric attribute measurer is responsible for evaluating the exact geometric relationships given specified geometric objects. The evaluated geometric relationships could be real/boolean values, or another geometric object. The real valued operators include veloci ty of (certain given object), location of (certain given object), (two objects against each other). The boolean valued operators include ins (detect intersection between two objects or an object and an auxiliary line), is moving, is www.ssssi.innnns.eeeeeo.oeeeeeee
weannceebtteen (two given objects),
(inspect one which returns the nearest
given object). Another special operator is fiidd.earest..obee class-í (i is given) object from a reference object in the current observation, and the attributes of this object could be further measured by other geometric attribute measurer. This operator is found be frequently used, as it is usually the case that when multiple objects are present, the agent tends to deal with the nearest object first
The numerical operator search space. In correspondence with the geometric search space, the numerical search space is designed for learning targets @ and θ, and its components are numerical operator combinations applied to the variables in the measured numerical state. There are also two types of numerical operators. The first type is the bool function/logiee operators, which include a, and conditional combinations: a a+I+ xyx The second type is the math operators, which include +, /,, sin, cos, tan, cot, (-)², (-) exp, log.
is(c < y). is( S y). is(c = y), I a
To illustrate with two concrete examples, in the environment CircusCharlit (illustrated Fig. [] and Fig. 2 the agent's target is jump right into the fire ring and over the pot. One sample distilled symboblic policy is presented in Fig 3 The equation [ allso represents a subtree of Fig 3 for simplicity. In Eq [ Fring pot and xo are the z-coordinate of the detected fire ring, pot and the protagonist objects. The distilled policy allows the agent to only jump when the condition -(10- < Topt Tring < 50)= x (Fring Fg < 40) is met. Pong (illustrated in Fig. I.. the agent needs to control the right racket so as to hit the pong, which can collide and bounce with the walls. One sample distilled symbolic policy is in Eq. 2 where Zpong is the x coordinate
pong, S is the intersection of the speed orientation and the right edge (drawn and tracked by the auxiliary line drawer). Such policy allows the agent to ignore the bouncing procedure of the pong, only pay attention to the speed orientation of the pong when it is near the edge (Ipong 40). More examples of distilled policies and their interpretations are in Appendix. C
One possible difficulty of the symbolic distillation procedure is the huge size of the search space: it grows exponentially with longer time span. Denote the number of symbolic compositions in one frame as N. If one consider stacking T' consecutive frames without duplicate removal or other engineering tricks, the total number of symbolic compositions will be N². To reduce the size of the search space, we primarily focus on single-frame policy distillation, i.c., infer the action only based on the current frame observation. For the environments that require multi-frame tracking, those
5

---

Under review as a conference paper at ICLR 2022
long- tergttermiii informations could be attached to the object as attributes of the object, and are carried and updated across time. For example, in the shooting based environments, whether an object has been set as the "next shooting target" by the teacher agent could be set attribute of this object, and this attribute could be measured by observing the teacher behavior, and backtrack in time to mark previous frames (backtrack is allowed during distillation, since our distillation algorithm works with the offline stored teacher behavior dataset), and could be presented as a component in the measured numerical state. Under this strategy, the size of the symbolic search space is greatly reduced, and plenty of environments are found to be viable for symbolic distillation (approximately 85% of 1000+ gym retro environments). A sampled list of such environments are put in Appendix [D]
policy-Cireuscharll = (10 < pot Fring < 50) x go_left +
-(10 < Tpot Tring < 50) x [(Tring - Io < 40) x jump+ (I)
(Xring To 40) go right]
policy pong = pFong > 40) x ((ypong us) x down + -(yoong > ys) x up) +
(2)
(xponn > 40) X nolaction
4.3 ROUNDTOURMIX: THE PROCEDURE To DISTILL THE SYMBOLIC POLICY Having established the search space and clarified the learning targets, in this section, we propose the RoundTourMix: gradient-free policy dis- tillation approach that iteratively optimize the three learning target discussed in Section 4.1 by switching back and forth between observ- ing teacher's behavior and interacting with the environments. It follows the straightforward philosophy of genetic programming: mainline guess of symbolic policy tree is maintained, new variants are repeatedly mutated from the main- line, and those better variants which minimize the loss with the teacher's 's actions are captured. We take the Cross- Cross-Entropy loss as the metrie under the discrete action space we tested. We also note the choice to use Mean Squared Error loss if the action space is continuous. Through this evolution procedure, the more complex and better performing mutations of symbolic policy are iteratively discovered.
NV
Figure 3: One sample distilled policy from a CNN agent in the CircusCharlic environment.
We note that the decision procedure of a wide range of CNN policy networks could be efficiently represented as high-fidelity tree shaped symbolic policy similar to Fig. 3 Therefore, we extend it into a generalizable form, where there are condition node and the action node co-existed. The condition node serves as the branching node, which corresponds to the learning target @, and the action node serves as the leaf node that corresponds to the learning target O
The condition node has three key properties: the condition, FLFFT, and GRIGHT. Given a sample of numerical state, the condition is a symbolic composition that evaluates into a boolean value. aLEFT and ORIGHT are two of its children tree nodes, which can be either condition nodes or action nodes. Given a numerical state, if the condition of this node is evaluated to be True, GLEFT will be executed, otherwise HRIHT will be executed.
The action node also have three key properties: the Lotal condition, the depth, and the policy. The total condition is the cascaded condition of its ancestral condition nodes, in other words, given any sample state, the total condition of an action node is true only if the condition of all of its ancestral nodes are met (either True or False), so that this action node can be reached along the path from the root node. The depth of the action node means the length of the path from the root node to the action node. The policy can either be a single action, or mapping from the state space (a subspace that already meets the total condition) to the action space. In practice, we encourage the policy to be single action, but if the tree search reached certain predefined maximum depth, we also allow this policy to be the symbolic mapping recovered by symbolic regression (Cranmer al. 2020 Cranmer 2020), which is a subtree itself.

---

Under review as a conference paper at ICLR 2022
Specifically, the Mou is composed of four mutually interactive stages:
Algorithm: Distilling Teacher Behavior into Symbolic Tree
Record Teacher Behavior. In the first stage, the teacher interacts with the environ- ment, and the visual observations are fully recorded and converted to geometrie symbols frame-by- saane--b-frame... We align these frames of symbols with actions and store as an offline teacher behavior datasel Do (no need to store teacher's reward). This stage serve the ini- tialization of RoundTourMis and is the only stage that will be executed only once.
Require: Temporary dataset Dirain containing
X (numerical states), Y (actions)
Return: r: the root of symbolic policy tree
Maintain: the of unsolved action nodes
1: Initializations
2:
r x- new Action NodeonNoddeeppplle = 0)
3: St (rr ; cnl <
4: While S { & ent < cntMAX:
5:
ent ent +1
6:
x (-pop(S > Sample action node
2. Guess and Observe. In this stage, we
7:
Yuu t [[n.total_condition >> Slices
make a random guess for the target geomet- ric relations be measured, then replay the teacher's ' interactions in Do- During the re- play, the numerical states are evaluated using geometric operators, and are recorded (instead
8:
IF Entropy(Yaau) sub) < Dentropy'
9:
n.policy f M eeann (ubb))
10: ELSE: D Single action cannot fit
11:
IF n.depth < depth.MA::
12:
With probability p1: & Split condition n. + newCondition.Node
of measure all possible geometric relations
13:
and yield gigantic numerical states). We col- lect the numerical states and teacher's actions
14:
S {n.ALEFT, n.nIGHH)
15:
With probability 1 - P1; D> De-noise
as a temporary dataset D, and split it into a training set Dirain and a validation set Deat-
16:
n.policy + default action
17:
ELSE: > Too deep, stop branching further
18:
With probability p2:
Solve. In this stage, we iteratively search
19:
Xanb + X(n.total condition]
the symbolic policy tree composed of the nu-
20:
n.policy t- -runSR(XX
Y sub)
merical operators. Table presents the details
21:
With probability P3: D -De-noise
of one round symbolic tree generation. This
22:
n.policy +- default action
procedure is repeated for multiple times, after
23:
With probability 1 1 p2 P3:
which a few of the best policy candidates on Deal are collected and sent to the next stage.
24:
< (ample(pathTTToooo((() remoucSubtree(n") n' 4 new.oondiiion
25:
26:
Verify and Optimize. In this stage, we let go the best few symbolic policies collected from last stage to interact with the environ- ment. If any candidate symbolic policy is found to match or surpass the teacher's s score, the RoundTourMix procedure is terminated and the distilled symbolic policy is found. Oth- erwise, return to the Guess and Observe stage and make new guess of which geometric relations to measure.
27:
S S+ Snn'-aa....T .ILEFT n' (aRIGHT)
28: Return r
Table 1: RoundTourMix's solve stage algorithm.
Remark 1: More Details in Stage 3. The core part of the algorithm in Table. D is to repeatedly guess branching conditions of the condition nodes (lines 13/26), slice out the subset of data that obey the total condition (line 7). and fit action nodes with this subset of data (lines 9/16/20/22). The more accurate the guessed condition, the easier it is fit action node. If the guessed condition happen to be near exact, the sliced teacher's is action tends to be nearly deterministic (i.c., low entropy, line 8, where Gentropy is threshold of the entropy). Otherwise, if the maximum depth depth m AX is met and the leaf node still is not deterministic (line 17), then it has the option to directly fit symbolic mapping (Cranmer 2020) (line 20).
Remark 2: Teaching Behavior Denoising. An important mechanism in this algorithm is the de- noising for the teacher behavior (lines 9/16/22). Under some conditions, the teacher model could possibly take noisy or "seemingly random" actions, which make it difficult for the RoundTouurMi to find a single deterministic action that matches well (line 8 not satisfied). In these cases, the distillation algorithm has the chance to assign the default action (line 16,22) denoise it. On other scenarios, the actions subset Ysub under certain total condition have low entropy but are not entirely deterministic, the symbolic distillation can still smooth it into deterministic single action (line 9), if the entropy is below threshold θ @entropy. Eventually, with the aid of the last screening stage of RoundTourrMx, the symbolic policy taking fewer but more efficient actions are picked the winner.
7

---

Under review as a conference paper at ICLR 2022
CromChurte
Porg Haar2603
Seusuesa 12000
12
1e0por Julti intertar
Tal Trwstegs- Soiesss
Tetal
The Trussess Toliee Dpiokss
A2C
Dted PJd
(a) Training performance
(b) Action distribution
Figure 4: Performance validation and behavior comparisons for the CNN based RL agents and the distilled symbolic rule. Action distribution (right) is for single evaluation round
Although this smoothing and denoising effect might potentially overlook some subtle behaviors of the teacher agent, we argue that the benefits of introducing such mechanism outweigh the disadvantages. On one hand, symbolic rules can still be encouraged to evolve into more complex trees to capture intricate teacher behaviors, c.g., by setting the depth of the tree controllable by depthMAx, and the richness/complexity of the symbolic building blocks, as defined by the search space. On the other hand, not all environments require intricate manipulations. For few environments that challenges on "balancing ability" (no need to involve much reasoning), such CartPole LunarLander, it is difficult to distill reliable symbolic policy. However, the visual reasoning tasks dominates the gym retro environments, and in these tasks, the denoised symbolic rules have demonstrated quite unbeatable competence across a host of them, as to be seen in section
5 EXPERIMENTAL SETTINGS AND RESULTS
In this section, we select several retro visual RL environments train CNN policy networks, then distill, verify and interpret the symbolic policy. Specifically, the selected environments include: AdventureIsland2, Adventurelsland3, Pong, CircusCharlie, Airstriker- Ainssssser-Genesis,, Seaquest, and As- troRoboSasa. More illustrative visualizations of our experiments can be found in the videos in the supplementary materials, and we sincerely recommend our readers could check them out.
Performance validation. We first compare the performance of the distilled symbolic policy against the RL. teacher models. Fig. Fig.Assoww shows the performance of PPO (Schulman et al.) (017). A2C (Miih al., 2016). human player, and the proposed symbolic policy distillation. In the experiments, the symbolic policies are all distilled from the PPO teacher model. The score of the human player is obtained by a gaming master controlling the agent under the lowered frame rate. Due to engineering facilitation considerations, in this work set the agents in the environments to only have one life for all. As can be seen in the figure, the distilled policy maintained comparable performance with the teacher model, and for some environments, the distilled policy is even better thanks to the denoising mechanism of symbolic distillation.
Interpretability validation. Besides verifying the performance, we also evaluate the interpretability of the regressed policy. This requires to check the symbolic operators combinations learned in the tree nodes, re-write them as readable form and analyze. On the selected environments, highly interpretable and hierarchical forms of symbolic policy trees are all found in the distilled equations. One sample distilled policy for Pong is displayed in Eq.2] and one sample for CircusCharlie is displayed in Fig. More interpretable distilled policies on other environments are put in appendix C
Behavior comparison with the neural network teacher. In Fig. 4 we plotted the action and the accumulated reward, both for the trained teacher model and the distilled symbolic policy. As can be seen in the figure, well-trained teacher model make more frequent actions, while still underperform
8

---

Under review as a conference paper at ICLR 2022
Evaluation on
Adventurelsland3
Adventurelsland2
Checkpoints from A1-3
10M
1.0M
2.5M
5.0M
7.5M
10M
Best
A2C PPO Distilled Symbolic Policy
5150
0
200
50
100
100
200
3000
150
100
50
50
50
3250
1950
Table 2: Policy Transfer comparisons. PPO and A2C are trained on Adventurelsland3 (AI-3), tested on Adventureisland2. The symbolic policy is distilled from PPO agent trained on AdventureIsland3. Neither PPO/A2C nor symbolic rules are iine-tnnddmmdiiied on Adventurelsland2.
the distilled symbolic rule. More differences are better shown in the video in the supplementary materials. The results there shows that in the Pong env, the trained CNN-based teacher model frequently "jitters" up and down even when the pong is very far away from the right racket. In those cases, the pong usually going to bounce with wall to change its heading direction, making those states "hardly controllable". Due the near-random behavior of the teacher under these "difficult" states, the symbolic distillation fails to converge to one single action under these cases, and hence mapped default action them, accidentally generated more efficient rules, and those rules picked through the last screening step of RoundToourMix.
Transferablity validation. Since the object detection step provide robust environment representations that group different scenarios together, even the scenes' pixel-level attributes are markedly dif- ferent, the distilled symbolic rule should transfer well if the logical specifications are congruent. In order to test the transferability/gener- alizability of the distilled symbolic rule, compare the the distilled rules against CNN policies using the AdventureIsland2 and Adven- tureisland3 environments (Refer Fig. 5). We first train teacher CNN policy networks on AdventureIsland3 and distill symbolic policy with PPO; we then directly test both the CNN models and the distilled symbolic rule on Adventurelsland? without tuning the CNN policy networks nor changing the symbolic rule.
0-02 Bo
As can be seen in Table when directly transfered new environ- ment with similar underlying rule but slightly different image styles, the CNN models completely fail to match their performances in the original environment, over the checkpoint from the entire training history. Additionally, fine-tuning the CNN policy network to adapt to the new environment will require another considerable amount of training. In contrary, the distilled symbolic rule painlessly transfer to the new environment, with significantly fewer performance loss. This is due to that our object detection module dwells on optical flow and object shape, which is robust color/style changes. The comparison between the symbolic policy and the CNN based policy network again endorses the better transferability of the symbolic rule.
Figure 5: Visual comparison
between the AdventureIsland2 (top) and AdventureIsland3 environments (bottom)
CONCLUSION
This paper study the distillation of CNN based reinforcement learning agent into symbolic policy that dwells on geometric and numerical operands and operators. Thanks to the the denoising and the screening mechanisms in the distillation procedure, the distilled symbolic policy achieves comparable or even better performance than the CNN teacher model. Our results point to the new opportunity to make the visual RL more interpretable, reliable and generalizable, by reclaiming symbolic design with the proposed symbolic policy distillation. Our future work aims for more integrated neural-symbolic solutions for visual RI.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY CHECKLIST
To ensure reproducibility, use the Machine Learning Reproducibility Checklist v2.0, Apr. 7 2020 (Pineau et al. 2021).
For all models and algorithms presented,
clear description of the mathematical settings, algorithm, and/or model. We
clearly describe all of the settings, formulations, and algorithms in Section 4.3
A clear explanation of any assumptions. We do not make assumptions.
An analysis of the complexity (time, space, sample size) of any algorithm. We do not make the analysis.
For any theoretical claim,
A clear statement of the claim. We do not make theoretical claims.
A complete proof of the claim. We do not make theoretical claims.
For all datasets used, check if you include:
An explanation of any data that were excluded, and all pre- pre-processing step. We did not exclude any data or perform any pre-processing.
A link to a downloadable version of the dataset or simulation environ- ment. All ROMs used can be downloaded from lattps://archi>>
detai http://eettaaassoooooooooooooooo 0011--00100000111110-00---111110 Fixee atari.
and
htttp://wwe a anttarimaan wimamiaa......mm.......aa...ii.m htt.a//iiaammmemmoooooeeeeoooiii 2600 roms ht.ml
For new data collected, complete description of the data collection process, such as instructions to annotators and methods for quality control. We do not collect or release new datasets.
For all reported experimental results, check if you include:
clear definition of the specific measure or statistics used to report results. We use the accumulated reward gained in single episode our statistical measure.
A description of results with central tendency (e.g. mean) & variation (e.g. error bars). We do not report mean and standard deviation for experiments.
-The average runtime for each result, or estimated energy cost. We do not report the running time energy cost.
A description of the computing infrastructure used. We do not report the computing infrastructure used.
REFERENCES
Osbert Bastani, Yewen Pu, and Armando Solar-Lezamal Verifiable reinforcement learning via policy extraction. arXiv preprint 1XX.v:800.00822 2018.
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc Le. Neural optimizer search with rein- forcement learning. In International Conference Machine Learning, pp. 459-468. PMLR, 2017.
Matthew Botvinick, Yael Niv, and Andew G Barto. Hierarchically organized behavior and its neural foundations: reinforcement learning perspective. Cognition, 113(3):222-280 2009.
Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Nowé, Tim Miller, Rosina Weber, and
Daniele Magazzeni. Distilling deep reinforcement learning policies in soft decision trees. In Proceedings of the IJCAI 20/9 workshop on explainable artificial intelligence, pp. 1-6, 2019.
Miles Cranmer. Pysr: Fast & symba symbolic regression in python/julia, September 2020. URL
2ttp:///2222oooo222222293 4052869
Miles Cranmer, Alvaro Sanchez-Gonzale Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. arXiv preprint arXiv:2006. /287, 2020.
10

---

Under review as a conference paper at ICLR 2022
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposi- tion. Journal of artificial intelligence research, 13:227-303, 2000.
Andrea Dittadi, Frederik K Drachmann, and Thomas Bolander. Planning from pixels in atari with learned symbolic representations. arXiv preprini arXiv:2012.09122 2020.
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. arXiv preprint rrxiv:9904. 1694, 2019.
Vincent Francois-Lavel Peter Henderson, Riashat Islam, Mare G Bellemare, and Joelle Pineau. An introduction deep reinforcement learning. Foundations and Trends@) in Machine Learning, 11 (3-4):219-354, 2018.
Artur Avila Garcez, Aimore Resende Riquetti Dutra, and Eduardo Alonso. Towards symbolic reinforcement learning with common sense. arXiv preprint arXiv:1804.08397, 2018.
Marta Garnelo, Kai Arulkumaran, and Murray Shanahan. Towards deep symbolic reinforcement learning. arXiv preprint 16Xvv:1609005118 2016.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In International Conference on Machine Learning, pp. 1792-1801. PMLR, 2018.
Piyush Gupta, Nikaash Puri, Sukriti Verma, Sameer Singh, Dhruv Kayastha, Shripad Deshmukh, and Balaji Krishnamurthy, Explain your move: Understanding agent actions using focused feature saliency. arXiv preprint 1rXiv:1912.12191, 2019.
Steven Gustafson, Edmund K Burke, and Natalio Krasnogor. On improving genetic programming for symbolic regression. 2005 IEEE Congress on Evolutionary Computation, volume 1, pp. 912-919. IEEE, 2005.
Andreas M Hein. Identification and bridging semantic gaps in the context of multi-domain engineering. In Forum on Philosophy, Engineering & Technology, pp. 58-57,000 2010.
Alexandre Heuillet, Fabien Couthouis, and Natalia Dodz-Rodr, Explainability deep reinforce- ment learning. Showledge-Based Systems, 21:: 106685, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 1003 2015.
Eddy 1lg. Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2462-2470, 2017.
Zhengyao Jiang and Shan Luo. Neural logic reinforcement learning. In International Conference on Machine Learning, pp. 31 10-3119. PMLR, 2019.
Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez. Explainable reinforcement learning via reward decomposition. In IJCAI/ECAI Workshop Explainable Artificial Intelligence, 2019.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model-based reinforcement learning for atari. arXiv preprint 1rXiv:1903.003:: 2019.
Daniel C Krawezyk. M Michelle McClelland, and Colin M Donovan. A hierarchy for relational reasoning in the prefrontal cortex. Cortex, 47(5):588-55)) 597, 2011.
Yitao Liang, Marlos C Machado, Erik Talvitie, and Michael Bowling. State of the art control of atari games using shallow reinforcement learning. arXiv preprint arXiv:1512.01563 2015.
Daoming Lyu, Fangkai Yang. Bo Liu, and Steven Gustafson. Sdrl: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 2970-2977, 2019.
11

---

Under review as a conference paper at ICLR 2022
Zhihao Ma, Yuzheng Zhuang, Paul Weng, Hankz Hankui Zhuo, Dong Li, Wulong Liu, and Jianye Hao. Learning symbolic rules for interpretable deep reinforcement learning. arXiv preprint arXiv:2103.08228, 2021.
Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through causal lens. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 2493-2500, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglon, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 132.5:131 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gym retro. https //g.thub.com//ppeaaal....
Santiago Ontanón, Kinshuk Mishra, Neha Sugandh, and Ashwin Ram. Case-based planning and execution for real-time strategy games. In International Conference on Case-Based Reasoning, pp. 164-178. Springer, 2007.
Jeff Orchard and Lin Wang. The evolution of a generalized neural learning rule. In 2016 International Joint Conference on Neural Networks (JCCNN, pp. 4688-4694, IEEE, 2016.
Brenden K Petersen, Mikel Landajuela Larma, T Nathan Mundhenk, Claudio P Santiago, Soo K Kim, and Joanne T Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. arXiv preprint arXiv: 19/204871 2019.
Joelle Pineau, Philippe Vincent-Lamare Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research. Journal of Machine Learning Research, 22:1-20 2021.
Thomas Philip Runarsson and Magnus Thor Jonsson. Evolution and design of distributed learning rules. In 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium Combinations of Evolutionary Computation and Neural Networks (Cat. No. 00, pp. 59-63. IEEE, 2000.
Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis, Sergey Levine, and Chelsea Finn. Reinforce- ment learning with videos: Combining offline observations with interaction. arXiv preprint arXiv:2011.06507, 2020.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data, science, 324(5923):81-88) 2009.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17770:1707.06347, 2017.
Pedro Sequeira and Melinda Gervasio. Interestingness elements for explainable reinforcement learning: Understanding agents' capabilities and limitations. Artificial Intelligence, 288:103367, 2020.
Wenjie Shi. Shiji Song, Zhuoyuan Wang, and Gao Huang. Self-supervised discovering of causal features: Towards interpretable reinforcement learning. arXiv preprint arXiv:2003.07069 2020.
Richard S Sutton, Doina Precup. and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 11((1-2):81122)) 1999.
Sabine Timpf, Gary S Volta, David W Pollock, and Max J Egenhofer. A conceptual model of wayfinding using multiple levels of abstraction. In Theories and methods of spatio-temporal reasoning geographic space, pp. 348-367. Springer, 1992.
12

---

Under review as a conference paper at ICLR 2022
Nicholay Topin and Manuela Veloso. Generation of policy-level explanations for reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2514- 2521,20199
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. International Conference on Machine Learning, pp.
PMLR, 2018.
Abhinav Verma, Hoang M Le, Yisong Yue, and Swarat Chaudhuri. Imitation-projected programmatic reinforcement learning. arXiv preprint arXiv: /e0p:09070553 2019.
Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. In International Conference on Machine Learning, pp. 1899-1908, PMLR, 2016.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement learning. arXiv preprint arXiv: 1000:008000000000 006.00830,0.00000000000011100008 2018.
A JUSTIFICATION FOR KEY NOTES
In this section we provide justifications for the architectural and algorithmic designs of our proposed RoundTourMix algorithm.
A.1 DETAILS ON THE SEARCH SPACE
Geometric and Numerical Operator Search Space. Precisely, we have a total of different functions in the geometric operator search space and 19 different functions in the numerical operator search space. As mentioned section
Temporal Tracking and Memory-Attributte. We include mechanisms that make our symbolic algorithm aware of the temporal behavior of its entities using memory-attribur shooting-based environments, relying just on the current frame gives rise to the agent shooting multiple times the same enemy if it is not aware of the enemies that it has already launched its bullet at. Another memory-attribure in such games arises when the agent needs to remember the target (out of multiple targets) it has locked to, get in suitable position and then fire.
For instance, in
A.2 MOTIVATIONS BEIIIND THE ALGORITIMIC CHOICES
In this section we break down the Step 2 and Step 3 of RoundTourMix while providing validation for the specific choices in the same.
Justification for Random Selection. Exhaustive search will ensure that excellent performance will be achieved with virtually unlimited hardware and software resources. However, as previously mentioned in 4.2 one potential challenge of the symbolic distillation technique is that the geometric representation's search space is exponentially huge. As a result, as compared to exhaustive search, random selection is the only viable option. Furthermore, search paradigm produces more reliable outcomes over exhaustive searching. we compose symbolic compositions of length 20, the total possible policies are (11 19)20 3020 3. 3.48e29 which is gigantic number. Furthermore, because there is no "learning" involved, any exhaustive or brute-force search goes against the spirit reinforcement learning. Traditional reinforcement learning does not use such comprehensive methods because of the vast number of viable policies, as earlier noted.
investigations show that the random
Guess and Observe. As earlier discussed the geometric search space is very large. Hence we randomly guess the 16 different geometric relations to be guessed. We then iterate over the image from the teacher behaviour, evaluate the randomly sampled geometric and numerical relations. For each frame we create a vector of size 16 of possible logics which are then later on used as the numerical state for symbolic distillation - The set numerical state vectors as extracted from each image constitutes the offline teacher dataset
Explanation of Branching Conditions.
13

---

Under review as a conference paper at ICLR 2022
Search the symbolic policy tree composed of numerical operators
Performed to get suitable candidate policies
We perform the search a total of cnt MAX times
At the start of each search we first sample an action node from the set all unsolved action nodes
We then slice out the subset of the offline teacher data that aligns with the guessed branching condition.
If the sampled branching condition was a step the right direction, the entropy of the subset sliced remains at a low value
If it does not, it an indicator that the underlying rule was more complex and the path is to be explored deeper
If the tree is already at its maximal depth, either (A) directly run and fit symbolic regression between the sliced numerical state vs action offline dataset (B) assign a default action to that action node or (C) prune off the entire sub-tree containing the action node in question
Else the tree not yet at its maximal depth and the entropy is high, we either (A) explore deeper by creating left/right branching condition or (B) assign default action at the action node in order to denoise it
B DETAILS OF OPTICAL FLOW BASED OBJECT DETECTION
Since in most retro environments, the image styles simple, where the backgrounds are nearly fixed and the objects share similar patterns. Therefore, we take the optical flow as simplified object detection module.
We first segment out the object list from template matching. then estimate the velocity using the pretrained FlowNet (Ilg et al. (2017), and get the speed via the average optical flow of detected objects. The classes of the objects are obtained based the shape in the predefined templates.
C EXAMPLES OF DISTILLED SYMBOLIC POLICY
We check the learned symbolic operators in the tree nodes, and re- write them interpretable symbolic tree policy. The results are plotted in Fig.
Fig. 7 Fig
10
Fig
Fig
and Fig
D EXAMPLES OF RETRO-ENV LIST THAT ONE-FRAME SYMBOLIC DISTILLATION CAN WORK WELL
After checking the underlying logics and evaluated the possible strategies of gym-retro environments, a plenty of them are found be solvable by our symbolic policy distillation. Several examples include: 8Eyes-Nes, 1942-Nes, BadDudes-Nes, BBioSenshiDanlnceeaaa Tonoo Tatakaaaaai--Nes BreakThru-- Nes, BubbleBobble-NNs, CodeName Viper-NNe, -ne GhostsnGoblins-Nes Kidlcarus-Nes, Wars-Ne,, Sansuu5/Anloo\ennecsaaCCim -eese
KidNikiRaadicllNinjae Parodius-Nes. TwinEagle-Nes, ViceProjectDoom Nes, Xexyz-Nes, Seicross-Nes, SonSon-Nes. Seenis-Atari26n MysteryQuest-Nes The screenshots for them displayed in Fig. [2
E
OTHER ILLUSTRATIONS
The pseudo-code for the algorithm in table are presented in Fig.
13
14

---

Under review as a conference paper at ICLR 2022
Start
Action Space
Jump
No Action
Charlie
Go Righh
Go Left
No
Right
type Ring?
Yes
No
<C
entity type
Is pearest
Pyst 3ydi
Yos
Yoe
Go Right
Zying zig
40
No
You
No
Jump
Right
Jump
Right
Figure 6: Distilled policy for CireusCharlie-Ner
15

---

Under review as a conference paper at ICLR 2022
IL Toaccing
NV
You
No Artion
attnded
Go flighti
Ceeaer
tnaking Tax
Left
smended in
Pube
Figure 7: Distilled policy for Eestriker-Genesee
16

---

Under review as a conference paper at ICLR 2022
Start
Up
Down
270 Tonn
40
Yes
No
No Action
No Action
Ypong yo
Yos
No
No Action
Ypong yo
Yes
No
Down
Up
Figure 8: Distilled policy for Pong-Atari2600
17

---

Under review as a conference paper at ICLR 2022
Hump
No Astio
TO
Raah
HEAR EpR
10
Rusei Ae
Jusige
Right
Right
IEEA Ipe
TOX
Fask
Jemp
Oh-Tihht
Action
Cs fiiht
Figure 9: Distilled policy for Adventureessandd-NNs
Star:
Up
Doinn
Playrr inooseet Plaze
Right
16
Lact
Cronet Ragh?
Closss
Abowt
Shoee
Righa
Shooe
Classt Point Tat??
No Action
BaTeon Tieesdy Abone?
Action
You
Sbuet
Figure 10: Distilled policy for AstroRoboSaaa-Nes
18

---

Under review as a conference paper at ICLR 2022
Tronkeg
30
TVC
abinded on
DOSO Flss
Tab diucte
Ratt
Figure 11: Distilled policy for Seaquest-Atari22
Figure 12: Several environments found be viable for symbolic distillation.
19

---

Under review as a conference paper at ICLR 2022
psudo-cade stage RoundTourtix
def solve_ hhlpppoollcccccceleeeeeeeecceeee sss_ssmbolic_treeee (ree(x, y):
input is 1ist of pains of teacher behavions:
numerical state
action
sysbalic with condition nodes action nodes initislize
www_actioo__nde(deppg
ansolved_action.
rost
lcop_cnt
while (unselved actionn action_nodes nodes is not expty) and (loop_ent cnt nax_cnt)):
loop_ent +*
node sample(unsolved, actionntttnn
randonly sarple an unsolved action node First check if the actions under the current total_condition is deterministic. "_subset y[node.total, tatootttalllooooe cond__oooiiion] select slices that satisfy cital_condition entropy(y_subset) entropyy_theeee threshold:
single
total_ condittoon, resolve
polecpolicy mean(y_subsett else:
1f nodt. nax depth:
depth is not net, branch on this node randonly guessed condition, and nark child nodes unsolved
htt_ooottooooodeeeeneeeee replace_actionn lat__aacctooonneeeoeoeeee aede__vitt_eeee netth_new_stt cacii_ee__ooomoe
unsolved, aac____cttonnnneeeee seciiinnnodeeeeeeee add[[ aaeeea,___L,,,ee_ aeteT,nodeaa_RReee
the current node is already too deep, then stop branching furthen. und unfform___1 rand() a sarple Fron unifors disteibtion [0,1] if unifoone_Bo 8_1 P_SR:
with probability P_5R, directly solve this node using Symbolic_.egressiinn.
s_subset conndd.total_contttee
polecy menbolic.Rerressionn....
y_subset)
untfors_a_1 a_1 P_SR p_dafault_ action:
with probability s_default_action set to default action to de-noise teacher behavior,
node.polscy defautt_action else:
Otherwise, renove subtree containing this node, than renew the searches. wode_father _father)
father new_condition_nekee
unsolved action mod_nooddsdddd aadd[[[__aathrr... father.a L_LE_,,,EE___
return root
Figure 13: The pseudo-code for the algorithm in table
20


Reviews:
[{"review_id": "x6Q8XA_qpzO", "reviewer": "Reviewer_cXsw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Review update**\nAfter reading through the authors' comments, I've decided to raise my score to a 6. I am willing to support acceptance of this paper. I believe it offers an original and definite contribution to an important but underdeveloped problem. It by no means solves all the problems need to bridge symbolic and black box deep RL, but having this work out there will allow others to build upon it.\n\n=========================\n\nThis was an interesting paper that shows the potential of symbolic RL, both in yielding better generalisation and in offering more human-interpretable policies. The authors' approach of distilling a black-box teacher policy into a symbolic one is quite interesting. The approach by no means bridges the gap entirely between black-box and symbolic RL, and the approach add some complexity by introducing extra hyperparameters including the maximum depth of the decision tree, an entropy threshold, and the set of symbolic building blocks that the symbolic policy can work with. The main strengths and weaknesses of the paper are as follows.\n\n**Strengths:**\n* original approach to distill black-box policies into symbolic ones\n* the symbolic policies found are human-interpretable\n* the symbolic policies are able to de-noise teacher policies, leading to qualitatively better policies (e.g. the symbolic agents take fewer harmless but wasteful actions at each time step; this means that we see, e.g., less paddle jitter in Pong when ball is far away, which we often see in PPO and other deep RL agents instead.) Perhaps the de-noising can help induce more efficient gaits in the Mujoco tasks.\n* the symbolic policies show superior transfer compared to their original teacher policies, showing the promise of symbolic policies in offering better generalisation\n\n**Weaknesses:**\n* some of the empirical results could be cleaner. This is the main weakness of the paper and is what makes me hesitate in fully recommending the paper for acceptance at this stage. Some of these issues, I believe, can be remedied by the authors during the discussion phase.\n    * how many random seeds were used for PPO, A2C and the distillation for all tasks? I see variances for PPO and A2C but not the distillation (which does use stochastic steps in e.g. Algo 1). It'd be helpful to know how robust is symbolic policy's superior performance to the choice of random seeds.\n    * the Atari results (Pong and Seaquest) are non-standard, making comparison with baselines in the literature difficult. First of all, they only allowed the agent a single life, and they train for 10M steps only (whereas 200M is more standard for published Atari results). Because of this, the paper's scores for PPO and A2C are far lower than what is in the literature.\n    * moreover, why is PPO’s learning curve for Pong far slower than that in Schulman et al, 2017 (https://arxiv.org/pdf/1707.06347.pdf). They seem to get near the max score of 20 points within 10M time steps.\n    * it would also be helpful to know whether the authors are using an open source release of PPO and A2C or a reimplementation. If it is an open source release, could the authors reference which implementations they used?\n* the authors gave a nice demonstration of the superior generalisation of their symbolic policies. The authors could also look to the Procgen Suite of games for further tasks to demonstrate generalisation. These were designed explicitly to train agents on one set of levels and test on held-out levels to evaluate agents' ability to generalise.\n* the authors mention tasks with continuous action spaces but offer no results on these (all reported results were on video games, which have discrete action spaces). What is the reason for this? If their approach was not as successful on continuous action tasks, it'd be helpful to report this and offer possible explanations why in the Discussion.\n* it'd be helpful to have some metric of the training complexity needed to distill a symbolic policy.\n* there were steps in the distillation algorithm that were unclear:\n    * what drives improvement in each iteration of Step 2 (Guess & Observe)? Particularly, what drives improvement of the geometric relations?\n    * how are candidate conditions for policy’s condition nodes chosen? It would seem like the list of possible conditions to pick is infinite, making search difficult.\n* the authors reference Botvinick et al (2009), but this reference seems completely irrelevant to the authors' discussion. The Botvinick et al paper is about hierarchical RL (hRL) in neuroscience. First of all, hRL does not seem relevant to what the authors are doing here -- hRL concerns hierarchical structure in policies where a higher level policy is defined in terms of macro actions, each of which defines a sub-policy in terms of primitive actions. This seems irrelevant to the authors' work here. Secondly, it is unclear why the authors chose to cite this particular hRL papers and not others, including Sutton, Precup, Singh (1999) on the options framework, Dietterich (1999) on MAXQ, Dayan & Hinton (1992) on feudal RL, or any of the many papers on deep hRL that have been published since.\n* minor point: the writing, while still highly readable, could benefit stylistically from the help of a proofreader more fluent in English. There were small grammatical errors (that did not hinder comprehension or ease of reading), but there were also awkward word choices. Here are just a few examples:\n    * p5, line 1 - “the learning target 1 rules the [geometric symbols→numerical state] step”: perhaps say \"1 determines the learning target for the [geometric symbols→numerical state] step\". Likewise for \"targets 2 and 3 rules the ...\" on the line below.\n    * p5, 2nd paragraph of the subsection \"The geometric operator search space\" - \"the nearest class-$i$ ($i$ is given) object from the **protagonist** in the current observation\": usually, \"protagonist\" refers to the main character of a story. Perhaps reword this to \"from **a reference object** in the current observation\"\n    * p6, last paragraph - \"father nodes\": the more common terminology is \"ancestral nodes\".\n", "summary_of_the_paper": "This paper proposes to bridge the gap between black box deep reinforcement learning (RL) and more interpretable symbolic RL by distilling learned deep RL policies into a symbolic decision tree. While the symbolic entities for each task are predefined, their algorithm is able to extract out a symbolic representation of the visual input as well as a symbolic policy in terms of a decision tree. The authors show that the distilled policy is highly human interpretable and even generalises better than the original teacher policy to tasks that are visually different but symbolically identical to the original training task.", "summary_of_the_review": "I believe this paper offers sufficient contributions to the field to potentially be accepted. Although it by no means solves all problems in this domain, it does help toward bridging the gap between black-box deep RL policies and more interpretable, symbolic RL policies. For me, the paper's primary contributions are the use of an original distillation procedure to translate black-box policies into symbolic ones, which offer the benefit of being more human interpretable and generalising better to tasks outside the training set. So this paper does have promise. But what prevents me from fully supporting it just yet are issues with the empirical evaluation. I would, in particular, like to ensure that the PPO and A2C baselines are accurately generated and that the results are robust to the choice of random seeds before I can fully support the paper's acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ZZKMJXYKsVY", "reviewer": "Reviewer_AL2N", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n* This paper is well-written and easy to follow. The proposed algorithm is also clearly explained and contains enough details to reproduce. \n* To my knowledge, the proposed method is a novel approach compared to prior works. In Figure 4, the distilled symbolic rule recovers and even occasionally outperforms the original teacher NN policy, which demonstrates the effectiveness of RoundTourMix. \n* I appreciate the increased interpretability of the distilled rules.\n\n**Weaknesses**\n\nWhile the results on Atari games and AdventureIsland are strong, I am primarily concerned with the applicability and effectiveness of the method on domains beyond toy 2D pixelated games. The paper makes many simplifying assumptions that may not hold in more complex and useful domains. To elaborate: \n\n1. Section 3 discusses the \"geometric symbol representation\" of the game pixels, which requires a semantic parser to preprocess the image. The exact approach (described in Appendix A) is template matching + FlowNet, which overfits to the toy game domain. This can be very brittle and result in compounding error in the downstream rule selection. Even worse, it will not be compatible with more realistic domains, such as:\n\n\t* 3D environment where the embodied agent can move freely and change perspective. The simple geometric shape parsing will fail, as the background changes dramatically with egocentric motion. \n\t* Occlusion, overlapping objects, deformable objects, etc. The current symbolic representation cannot handle any of these complex scenarios. \n\t* Locomotion tasks with a complex robot morphology that is unclear how to parse from pixels. \n\nMeanwhile, blackbox approach like PPO will be able to handle all of the above without the hand-engineered preprocessing pipeline. \n\n2. Section 4.2 discusses the geometric operator search space, which contains elements like `velocity_extension` and `static_line_drawer`. These are hand-engineered primitives tailored for simple 2D environments that cannot be easily applied to more sophisticated tasks. For example, how to handle complex contact interactions between the robot and its surroundings? What about deformable objects? Objects that change state (like cooking tasks)? Navigation in 3D world? Mobile manipulation? Once again, approaches like PPO can adapt to all these tasks _without case-by-case modifications_. This severely limits the usefulness of the proposed symbolic rule system. \n\n3. The symbolic rules do not seem to have any mechanism for long-term memory. How can it be applied to tasks that require long-horizon planning and reasoning, rather than short-term \"muscle memory\"? PPO + LSTM solves this out of box. \n\n4. The \"transferability validation\" experiment in section 5 gives an unfair advantage to rule-based policy, because it makes the strong assumption that there exists a perfect object recognition system for all game levels, even with very different visual appearance. This certainly holds for AdventureIsland, but does not hold for any real-world tasks. The same invariance can be achieved for ConvNet if we pretrain the early layers of an NN policy to be domain-agnostic, using techniques like data augmentation, domain randomization, or contrastive representation. \n\n5. The authors claim that \"smoothing and denoising\" help make the distilled rule more robust, and show the performance boost in Fig. 4 for certain tasks. However, similar \"denoising\" techniques can be applied to PPO as well, in the form of robust representation learning. For example, the following works all contribute to better neural representations and more robust policies. It is unclear if symbolic rules can consistently outperform these approaches:\n\n    * Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels. Yarats et al. ICLR 2021. \n    * Reinforcement Learning with Augmented Data. Laskin et al. NeurIPS 2020. \n    * CURL: Contrastive Unsupervised Representations for Reinforcement Learning. Srinivas et al. \n    * Learning Invariant Representations for Reinforcement Learning without Reconstruction. Zhang et al. ICLR 2021.\n\nFurthermore, I strongly disagree with the general claim that \"our symbolic policy distillation approach captures the causal structure through data-driven experience\" (introduction section). \n\nFirst, the symbolic rule distills from an NN trained policy, which necessarily means that it will inherit any spurious correlation learned from the teacher. Even though the authors introduce tricks like \"denoising\", it will not be able to recover the true causal structure without any explicit mechanisms like do-calculus or counterfactual reasoning. \n\nSecond, the learned symbolic rules develop hard-coded constants that are _also very brittle_. For example, what if the Pong game is slightly stretched and the `X_pong` coordinate threshold is 45 instead of 40? Humans will have no problem generalizing, but the symbolic rule will fail even though the causal mechanism of the game remains the same (and visual appearance is almost identical). ", "summary_of_the_paper": "This paper introduces an algorithm called RoundTourMix to distill a trained neural network policy into an executable symbolic form. The algorithm is gradient-free and denoises the original end-to-end policy. Results are shown on Atari games and AdventureIslands, which demonstrate that the distilled symbolic rules retain the original performance level while being more resistant to domain shift. ", "summary_of_the_review": "While the paper is overall clear and easy to follow, it makes many strong assumptions that are tailor-made for simple 2D games. It is unclear how the symbolic rule framework can scale to more complex domains. I am unconvinced that the distilled rules are more robust or causal than the PPO counterpart, especially when trained with robust representation learning techniques. \n\nI give my rating due to the limited usefulness and problematic assumptions of the proposed approach in realistic tasks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "S0SV8NzL4Uj", "reviewer": "Reviewer_wezQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper can be hard to follow at times, and could be simplified and clarified (e.g. 4.1 could be a figure, 4.2 could benefit from adding one). Another example is that Step 2 \"Guess and Observe\" of RoundTourMix is not defined in enough details (how many geometric features? picked how?) for the results to be reproduced. \n\nThere is a bit of missing related work. Specifically, \"State of the Art Control of Atari Games Using Shallow Reinforcement Learning\" (Liang et al. 2015) that trains linear models on higher level (almost symbolic) features extracted similarly as in this paper. In fact, those features are from the origin of Atari Learning Environment. A similar work is \"Planning From Pixels in Atari With Learned Symbolic Representations\" (Dittadi et al. 2020). Additionally, the litterature of inductive logic programming and case-based reasoning applied to (video) games is relevant, as inferring case-based policies from demonstrations has been an active area of research (e.g. \"Case-Based Planning and Execution for Real-Time Strategy Games\" (Ontanon et al. 2007)).\n\nThe experimental results show that a more robust policy than the PPO and A2C learned policies gets extracted for most of the subset of games that the authors picked (also exemplified in Figure 4.b.). But, this does not necessarily justify the choices of the RoundTourMix algorithm (central contribution of the article). Indeed there is no justification why one has to make a random selection (2. Guess and Observe) of the geometric relations, and why we can't simply be exhaustive for those games (I believe we can at least for Pong and Seaquest). This is my biggest criticism of this paper: there is very little justification and/or empirical validation for the many choices of the RoundTourMix heuristic. \n\n\"Transferability validation\" is an interesting experiment, that shows that the symbolic policy (distilled from a CNN trained with PPO on AdventureIsland 3) transfers to a similar but different game, AdventureIsland 2. Sadly, there was no apples to apples comparison by having a PPO trained policy simply plugged onto the geometric symbol representation. This would have allowed to show how much of the transferability comes from the features and how much comes from the \"if-this-then-that\" policy.", "summary_of_the_paper": "The paper presents a method for distillating potentially any policy into a symbolic policy (in prepositional logic + basic maths operations), applied in 2D video games based on the Atari, NES, and Genesis emulators. The procedure (RoundTourMix) consists in applying random search to iteratively i) select the features of interests, and ii) solve for a symbolic policy (finding conditions / FSM induction) that works with those features. The teacher policy in all experiments in a neural network trained with PPO. The conversion from pixels to objects (\"geometric features\") is assumed solved. Experimental results are presented on 7 games.", "summary_of_the_review": "The central contribution of the paper (the RoundTourMix distillation procedure) seems ad-hoc and many of its decisions are not well-enough justified. The experimental validation has some flaws that need to be addressed to make it a convincing contribution.\n\nFinally, this is honestly not a great fit (topically) for ICLR and would be better suited in a games (e.g. AAAI AIIDE, IEEE CoG) conference or workshop. In its current form, the contribution is too \"heuristic-y\" for publication at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7mDYgiR4UI", "reviewer": "Reviewer_GGAU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- To my knowledge this is a novel algorithm that shows promising results on a small number of games and shows to be more robust to changes in the environment.\n- Resulting decision making is easily interpretable by humans\n- The work is very clearly presented\n\nWeaknesses\n- Minor: “Abstracting pure symbolic rules from data is not new. The family of symbolic regression (SR) methods (Cranmer et al., 2020; Runarsson & Jonsson, 2000; Gustafson et al., 2005; Orchard & Wang, 2016) have been studied to to directly.” Remove double “to”.\n- Being a purely offline learning method it seems like the maximum possible performance of this method is closely related to the expert policy and the types of levels that training experience is gathered on. It would be very interesting to know whether the authors have thought about whether their proposed method could be extended to learn from online data generated by itself.\n- Another concern is that thinking about their proposed method, it seems that the method thrives on levels where the optimal action can be inferred from the current observation and that a predominantly reactive policy can perform well. I am curious about how their method would perform in a game like sokoban that requires careful planning, or others, where the agent needs to remember information that is no longer visible in the current observation. It seems to me that their proposed symbolic method does not have any memory and would therefore not achieve similarly impressive results.\n", "summary_of_the_paper": "The paper presents a novel algorithm that distills expert experience in the Gym Retro Games environment into an interpretable and symbolic policy. The main steps in their method are the following: first gather experience from a trained network and pre-process with pre-trained and frozen networks to extract a list of objects, their type, position and velocity for every image observation. Then these are passed to the next module that computes numerical states, such as auxiliary lines between the objects and computing relations between the objects. Finally, their novel algorithm makes use of genetic mutation to discover rules that consume these relations and take actions. The last part is trained to match the data from the expert and only then deployed for evaluation in the environment. Their empirical results show that their method can perform well and even outperform the expert policy used to collect the initial experience. Moreover, they show that their proposed method transfers much better compared to the end-to-end baselines (PPO and A2C) from AdventureIsland3 to AdventureIsland2, making it a more robust method. Importantly, they show how easy it is to interpret the decision making that selects the action from the current state.", "summary_of_the_review": "I recommend a weak acceptance for the paper, due to the novelty of their algorithm and the interesting empirical results presented, where the final policy is very interpretable. My main concerns are with the wider applicability of the method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4DJg3AB6HW1", "reviewer": "Reviewer_VSj4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The high-level procedure is relatively straightforward. Train a DRL agent directly on the environment, and then, given symbols extracted from each state, \"distill\" the policy into a symbolic policy (note that supervised learning applied in the context of imitation learning is known as behavioural cloning [1], and what is being done here also falls into the category of imitation learning/learning-from-demonstration). It is difficult to find symbolic policies from scratch, so this procedure makes sense, and results in policies that are (generally) more interpretable and generalise better. Without knowing the literature on symbolic regression/symbolic RL too well, to the best of my knowledge the technical contributions of this work are very good.\n\nA weakness is the strong reliance on object detection and geometric priors for extracting/forming symbols. As it is, the algorithm cannot be applied to more complex visual domains. However, I think we should not discourage research along these directions, as the general idea of distilling black-box policies into symbolic policies has good use cases.\n\nSome questions:\n- How important is the entropy threshold? Any way for setting/tuning this automatically?\n- Are there any (quantitative) ablation studies on the denoising procedure? It would be useful to know if the algorithm fails without this or just has worse performance.\n\nConsidering that the authors' method incorporates imitation learning, they may find methods such as DAgger [2] of interest. Assuming access to an expert policy that can be queried interactively, interactive imitation learning methods benefit from reducing the shift between the teacher's state-action distribution and that of the student (which is a known problem in behavioural cloning, and is potentially an issue for this method as well).\n\n[1] Pomerleau, D. A. (1989). Alvinn: An autonomous land vehicle in a neural network. CARNEGIE-MELLON UNIV PITTSBURGH PA ARTIFICIAL INTELLIGENCE AND PSYCHOLOGY PROJECT.\n[2] Ross, S., Gordon, G., & Bagnell, D. (2011, June). A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics (pp. 627-635). JMLR Workshop and Conference Proceedings.", "summary_of_the_paper": "The authors propose a novel method for finding symbolic policies for image-based RL environments. The method consists of training a traditional DRL algorithm, and then \"distilling\" this policy by searching for a symbolic policy that mimics the DRL teacher agent. This can result not only in interpretable, lightweight policies, but also policies that generalise better than the original teacher policy.", "summary_of_the_review": "I believe the authors tackle an interesting and relevant problem in RL, combining both deep learning and symbolic approaches to good effect. To the best of my knowledge, there are solid contributions in this work, and I would recommend accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: After going over the reviews and the rebuttal, and skimming the paper, I feel like unfortunately this paper is not ready to be accepted.

My reasoning is as follows. I feel the comparison with A2C and PPO is not and should not be the main target of the work. Of course they are good to have as reference points, and they should be in the paper. But the work is not trying to claim that the distilled symbolic policy is more data efficient (or outperforms these methods). If that would be the point, that one has questions similar to reviewer cXsw about these baselines maybe underperforming (compared to other published work). Maybe this is due to a change in setup as argued by the rebuttal, nevertheless this makes comparison and understanding the results difficult. The other argument is that  A2C / PPO are not the most efficient DRL methods for atari.  Lastly, is the question of distilled symbolic policy having access to an expert, making this not an apples to apples comparison. 
But as I said, and I think this is the point of the authors as well, this is not the point of the paper. But then I find the results not being sufficiently contextualized either by comparing to other methods in this space, or various ablation studies to motivate the choices taken by the authors. Similar points were raised by other reviewers (wezQ, cXsw). Some of these ablations have been brought forward in the rebuttal, but I think they should be a more central part of the work and implies considerable edits to the paper. 
I think the stance that the object identification is decoupled from the symbolic policy is also a bit dangerous. I.e. a learned object identifier (particularly in a visual more complex setting) will have different failure modes, which will affect the policy. I think having a paragraph discussing the issues raised by reviewer AL2N would actually strengthen the paper, and being open about open questions/weaknesses. Alternatively additional ablation or experiments in either other kind of environments (e.g. 3D or environments with occlusion) or just assuming some form of failure at segmentation the visual stream into objects to show robustness would be of interest. 

Overall I urge the authors to resubmit their work after properly integrating some of the feedback. In particular focusing on ablation studies or having baselines that are more similar in spirit or at least being more explicit of how it compares with existing work and what aspect of that existing work is trying to fix. For e.g. part of the approach is that it relies on distillation rather than dealing with the RL objective (as other methods might try to do). Now if you take those methods, but you phrase them in a distillation process how would they do? I don’t know if all of this needs to be done, but it just feel as a work to be less grounded and sufficiently far to other existing methods to trivially understand the relationship, while directly only comparing to non-symbolic methods in a way that is not in some sense in the advantage of the non-symbolic methods. 
 Additionally,  being more explicit about the potential weaknesses of the method, maybe empirically showing what happens with imperfect segmentation. The work is interesting, and I agree that this is a young field and the goal is *not* to produce state of the art results or outperform DRL methods. And is *not* to solve all the problems with symbolic methods at once, but to improve our understanding in this space. But I think the framing is not the right one in the current manuscript.
Decision: Reject