Mitigating Information Loss in Tree-Based Reinforcement Learning via Direct Optimization
Authors: Sascha Marton
Tim Grams
Florian Vogt
Stefan Lüdtke
Christian Bartelt
Heiner Stuckenschmidt
Venue: ICLR 2025 Spotlight (2025)
Keywords: Symbolic Reinforcement Learning
Interpretable Reinforcement Learning
Reinforcement Learning
Decision Trees
Policy Gradient
Proximal Policy Optimization
Primary area: reinforcement learning
TL;DR: SYMPOL is a novel method for symbolic RL that enables end-to-end gradient-based learning of interpretable, axis-aligned decision trees, combining policy gradient optimization with symbolic decision-making

Abstract: Reinforcement learning (RL) has seen significant success across various domains, but its adoption is often limited by the black-box nature of neural network policies, making them difficult to interpret. In contrast, symbolic policies allow representing decision-making strategies in a compact and interpretable way. However, learning symbolic policies directly within on-policy methods remains challenging.
In this paper, we introduce SYMPOL, a novel method for SYMbolic tree-based on-POLicy RL. SYMPOL employs a tree-based model integrated with a policy gradient method, enabling the agent to learn and adapt its actions while maintaining a high level of interpretability.
We evaluate SYMPOL on a set of benchmark RL tasks, demonstrating its superiority over alternative tree-based RL approaches in terms of performance and interpretability. Unlike existing methods, it enables gradient-based, end-to-end learning of interpretable, axis-aligned decision trees within standard on-policy RL algorithms. Therefore, SYMPOL can become the foundation for a new class of interpretable RL based on decision trees. Our implementation is available under: https://github.com/s-marton/sympol

Published as a conference paper at ICLR 2025
MITIGATING INFORMATION LOsS IN TREE-BASED RE- INFORCEMENT LEARNING VIA DIRECT OPTIMIZATION
Sascha Marton 1*
Tim Grams
Florian Vogt!
Stefan Lüdtke
Christian Bartelt
Heiner Stuckenschmidt'
University of Mannheim Technical University of Clausthal Uniiersity of Rostock
sascha ma rtonrnnnnnnnn.oo maa.dmiimmmmnneimmmccm tim.n noc. graamstu-cleee .de
ABSTRACT
Reinforcement learning (RL) has seen significant success across various domains, but its adoption often limited by the black-box nature of neural network poli- cies, making them difficult to interpret. In contrast, symbolic policies allow repre- senting decision-making strategies in compact and interpretable way. However, learning symbolic policies directly within on-policy methods remains challenging. In this paper, we introduce SYMPOL, a novel method for SYMbolic tree- ree-based on-POLicy RL. SYMPOL employs tree- rree-based model integrated with a policy gradient method, enabling the agent to learn and adapt its actions while maintain- ing a high level interpretability. We evaluate SYMPOL on a set of benchmark alternative tree-based RL approaches
RL tasks, demonstrating its superiority in terms of performance and interpretability. Unlike existing methods, it enables gradient-based, end-to-end learning of interpretable, axis-aligned decision trees within standard on on-policy RL algorithms. Therefore, SYMPOL can become the foundation for new class of interpretable RL based on decision trees. Our imple- mentation is available under: https:// /ttp://ggtttttt///// com/s marton/ /smp//
-203
-403
-000
-003
-3000
-200
-003
-3000
-000
-083
-3000
S0HCCL Tamn
SYNOOL Tut
-2000
NR. Tuan
54.07 Teat
507 Rain
0-5OT Text
5004 (a) SYMPOL (ours)
203c
400c
3004
203c
483c
683s
063
2006
400
600
14
State-Action DT
(c) Discretized Soft DT
Figure 1: Information Loss in Tree-Based Reinforcement Learning on Pendulum. Existing methods for symbolic, tree-based RL (Figure Ib and Icp suffer from severe information loss when converting the differentiable policy used for training (e.g.. the MLP for SA-DT) into the symbolic policy used for interpretation (i.e., the DT). Using SYMPOL (Figure Tap, we can directly optimize the symbolic policy with PPO and therefore have no information loss during the application.
INTRODUCTION
Reinforcement learning lacks transparency. Reinforcement learning (RL) has achieved remark- able success in solving complex sequential decision-making problems, ranging from robotics and autonomous systems to game playing and recommendation systems. However, the policies learned by traditional RL algorithms. represented by Neural Networks (NNs), often lack interpretability and transparency, making them difficult to understand, trust, and deploy in safety-critical or high-stakes scenarios (Landajuela al. 202T
"Equal Contribution

---

Published as a conference paper at ICLR 2025
Symbolic policies increase trust. Symbolic policies, the other hand, offer promising alterna- tive by representing decision-making strategies in terms of RL policies as compact and interpretable structures (Guo et al., 2024). These symbolic representations do not only facilitate human under- standing and analysis but also ensure predictable and explainable behavior, which is crucial for building trust and enabling effective human-AI collaboration. Moreover, the deployment of sym- bolic policies in safety-critical systems, such as autonomous vehicles or industrial robots, could significantly improve their reliability and trustworthiness. By providing human operators with clear understanding of the decision-making process, symbolic policies can facilitate effective moni- toring, intervention, and debugging, ultimately enhancing the safety and robustness of these systems. In this context, decision trees (DTs) are particularly effective as symbolic policies for RL, as their hierarchical structure provides natural interpretability.
Existing challenges. Despite these promising prospects, the field of symbolic RL faces several challenges. One main reason is given by the fact that many symbolic models, like DTs, are non- differentiable and cannot be integrated in existing RL frameworks. Therefore, traditional methods for learning symbolic policies often rely on custom and complex training procedures (Costa et al.)
2024 Vos & Verwer 2023 Kanamori et al.) 2022), limiting their applicability and scalability. Al- ternative methods involve pre-trained NN policies combined with some post-pri post-procsssing to obtain
an interpretable model (Silva et al.] 2020 Liu et al.] 2019 2023 Bastani et al. 2018). However, post-processing introduces a mismatch between the optimized policy and the model obtained for interpretation, which can lead to loss of crucial information, as we show in Figure
Contribution. To mitigate the impact of information loss, direct optimization of the policy is crucial. Existing methods that employ a direct optimization of a tree-based policy, such as those described by Silva el al.] (020)) learn differentiable, soft decision trees which do not provide a high level of interpretability. To obtain interpretable, axis-aligned DTs, these methods require post-hoc distillation or discretization and therefore suffer from information loss (see Figure T]. In this paper, we introduce SYMPOI., SYMbolic tree-based on-POLicy RI., a novel method emplaying direct optimization axis-aligned DT policies end-to-end. Our contributions are as follows:
We integrate GradTree (Marton et al. 2024a) into existing RL frameworks via a separate actor-critic architecture to directly optimize DT policies and extend continuous action spaces (Section A.1)
We propose a dynamic rollout buffer to enhance exploration stability and a dynamic batch size through gradient accumulation to improve gradient stability (Section 4.2) to mitigate the instability of DT training in dynamic environments.
We propose using weight decay on a subset of parameters to support a dynamic adjustment the model parameters when optimizing DTs with gradient descent (Section 4.1)
As a result, SYMPOL does not depend on pre-trained NN policies. complex search procedures, or post-processing steps, but can be seamlessly integrated into existing RI. algorithms (Section 3)
Results. Through extensive experiments on benchmark RL environments, we demonstrate that SYMPOL does not suffer from information loss and outperforms existing tree-based RL approaches in terms of interpretability and performance (Section 5.2). providing human-unddrssanddbbl expla- nations. In most environments, SYMPOL's performance is comparable to full-complexity models, while in categorical environments, even surpasses them. Furthermore, we provide a case study (Section 6) to show how interpretable policies help in detecting misbehavior and misgeneralization which might remain unnoticed with commonly used black- box policies.
2 RELATED WORK
Recently, the integration of symbolic methods into RL has gained significant attention. Symbolic
RL does cover different approaches including program synthesis (Trivedi et al., 2021 Penkov &
Ramamoorthy] 2019 Verma et al., 2018). concept bottleneck models (Ye et al. 2024). piecewise
linear networks Wabartha & Pineau| (024) and mathematical expressions (Landajuela et al. 2021
Guo et al., 2024 Luo et al. 2024 Kamienny al., 2022). Another line of work aims to synthesize
symbolic policies using logical rules, leveraging for instance differentiable inductive logic program-
ming for gradient-based optimization (Delfosse et al. 2024b Jiang & Luo 2019 Cao al. 2022
2

---

Published as a conference paper at ICLR 2025
In contrast to first-order rules, DT's offer greater flexibility by not only combining atomic conditions but also comparing features against thresholds - a critical capability for handling continuous ob- servation spaces. Furthermore, trees have also been in used in other agentic components than the policy, such as reward functions (Milani et al. 2022 Kalra & Brown] 2023 2022). In this paper, focus exclusively
tree- tree-based based methods for symbolic RL. Similarly, ensemble methods (Fuhrer) it al.] 2024] Min & Elliot| (022) have been proposed. However, policies consisting of hundreds of trees and nodes lack interpretability and therefore are out of scope for this paper. Several approaches have been proposed to leverage the strengths of interpretable, tree-based representations within RL frameworks, each approach comes with its own critical limitations. We summarize existing methods into three streams of work:
(1) Post-processing. One line learns full-complexity policies first and then performs some kind of post-processing for interpretability. One prominent example is the VIPER algorithm (Bastani al. 2018) where a NN policy is learned before distilling a DT from the policy. However, dis- tillation methods often suffer from significant performance mismatches between the training and evaluation policies (Figure [bb.. To mitigate this mismatch, existing methods often learn large DTs (VIPER learns DTs with 1,000 nodes) and therefore aim for systematic verification rather than in- terpretability. In contrast, SYMPOL is able to learn small, interpretable DTss (average of only 50 nodes) without information loss. Following VIPER, various authors proposed similar distillation methods (Li et al.) 2021; Liu et al., 2019, 2023: Jhunjhunwala et al., (020). Furthermore, KKohee al. (2024) propose a novel method that distills interpretable and editable programmatic tree poli- cies. In contrast to SYMPOL, the extracted trees are not considered axis-aligned, as they allow for linear combinations and multiple features within the internal nodes.
(2) Custom optimization.
Methods involving custom, tree-specific optimization techniques
and/or objectives (Ernst et al., 2005) Roth al. 2019 Gupta et al.] 2015) Kanamori et al., 2022)
are generally more time-consuming and less flexible. As a result, their policy models cannot be
easily integrated into existing learning RI. frameworks. Examples are evolutionary methods (Costa
et al.
2024 Custode & lacca| 2023) and linear integer programming (Vos & Verwer] (023). TTopin
ict al. 2021) propose Iterative Bounding Markov Decision Process (IBMDP) that allow learning
DT policies through masking procedure and modified value updates by using arbitrary function approximators. However, using IBMDP, the learning problem becomes more complex compared to the base MDP, which can result in poor scalability and limits the applicability very simple tasks (Milani et 2022 Kohler et al. 2024). In contrast, SYMPOL optimizes a DT policy directly on the base MDP, avoiding these limitations.
(3) Soft Decision Trees (SDTs).
Methods optimizing SDTs (Silva et al. 2020 Silva & Gom-
bolay 202T Coppens et al. 2019: Tambwekar al., 2023 Liu et al.) 2022 Farquhar et al. (017) are difficult to interpret since they usually involve multiple features simultancously at each decision node, creating complex, multidimensional splits rather than straightforward, single-feature thresh- olds. Nevertheless, the trees are usually not easily interpretable and techniques such as discretizing the learned trees into more interpretable representations are applied (Silva et al. 2020)) occasionally resulting in high performance mismatches (Figure Tc). In contrast, SYMPOL directly optimizes hard, axis-aligned DT's and therefore does not exhibit performance loss (Figure Ta).
Distinction of SYMPOL from Differentiable and Soft Decision Trees. In existing work like Silva et al. (020). differentiable decision trees typically correspond to SDTs, achieving differen- tiaiility by relaxing discrete decisions in terms of feature selection at each internal node and path selection. This approach is fundamentally different from SYMPOL, which does not use differen- tiable decision trees. Instead, SYMPOL leverages GradTree to optimize standard, non-differentiable decision trees through gradient descent, as we will show in Section 4
PRELIMINARIES
Markoy Decision Process (MDP). We study a deterministic MDP (S, (S,,,,,,,)) where and A are finite state and action spaces, P: SxAxS - [0. 1] defines the transition dynamics, r: SxA + IR is the reward function and y the discount factor. At each timestep t, an agent samples an action o from policy œ! S + A based on observation s, S and executes it, receiving reward r,. The value function (T)(s) = Ea.wr, 1 3'r(s,. 4x) [ st = s) approximates the expected return when starting in state s and then acting according policy æ. Similarly, the action-value function
3

---

Published as a conference paper at ICLR 2025
En= ? 8, 8, a] estimates the expected return when selecting action a in state s and then following policy я. Finally, the advantage function A (s, a) = (", a) VT (s) defines the difference between the expected return when choosing action in state 8 and the expected return when following the policy æ from state 8. Overall, we aim for finding an optimal policy x² that maximizes the expected discounted return [xz) E [Er= 7'r(st, ae)].
(" a) Bainaas
TMP 1
Proximal Policy Optimization (PPO). PPO (Schulman et al. 2017) is an on-policy, actor-critic
RL method designed enhance the training stability. The algorithm introduces clipped surrogate objective to restrict the policy update step size. The main idea is to constrain policy changes to a small trust region, preventing large updates that could destabilize training. Formally, PPO optimizes:
(at ssc) Ac, clip To (ax|se)
<o(ae(bs)) Tow (ar)se))
CLP =Bar
min
(
(I)
where xo(ails) Ty (ar) is the probability ratio between the new policy Te and old policy Tewu Ag is an estimate of the advantage function time step / and is hyperparameter for the clipping range.
SYMPOL: SYMBOLIC ON-POLICY RL
In the following, we formalize the online training of hard, axis-aligned DT's with the PPO objective. Therefore, SYMPOL utilizes GradTree (Marton et al. 2024a) as a core component to learn a DT policy directly from policy gradients. In contrast to existing work on RL with DTs, this allows an op- timization of the DT on-policy without information loss. The main conceptual difference to existing work that learn symbolic policies end-to-end (Delfosse et al. 2024b Fuhrer et al., 2024 Delfosse et al. 2024c Topin et al. 202T Luo et al.) is that SYMPOL does not require any modification of the RL framework itself. making the proposed method framework -agnosst.....sticc. As result, interpretable policies with SYMPOL are learned in the same way as NN policies are commonly learned. In the main paper, we focus on PPO as the, we believe, most prominent on-policy RL method. To support claim of seamless integration, we provide additional results using Advantage Actor-Crite CCittC (A2C) (Mnih et al. 2016) in Appendix AA. To efficiently learn DT policies with SYMPOL, we employed several crucial (see ablation study in Table modifications, which we will elaborate below.
4.1 LEARNING DT's WITH POLICY GRADIENTS
Arithmetic DT policy formulation. Traditionally, DTs involve nested concatenations of rules. In GradTree, DTs are formulated as arithmetic functions based on addition and multiplication facil- itate gradient-based learning. Therefore, our resulting DT policy is fully-grown (i.e., complete, full) and can be pruned post-hoc. Our basic pruning involves removing redundant paths, which signifi- cantly reduces the complexity. We define path as redundant if the decision is already determined either by previous splits or based on the range the selected feature. More details are given in Appendix A.4 Overall, we formulate DT policy of depth d with respect to its parameters as:
x as
Σ (
where L is a function that indicates whether state s E R belongs to leaf I, a E - denotes the selected action for each leaf node, 7 E R22-1 represents split thresholds and E N22 the feature index for each internal node.
Dense architecture. To support a gradient-basee optimization and ensure an efficient computation via matrix operations, we make use of dense DT representation. Traditionally, the feature index vector & is one-dimensional. However, as in GradTree, we expand it into a matrix form. Specif- ically, this representation one-hot encodes the feature index, converting (22 -)x(s|) Similarly, for split thresholds, instead of single value for all features, individ- ual values for each feature are stored, leading to T E (=22-1)x
R²"-1 into matrix
The dense representation is visualized in Figure 2 Please note, that in contrast SDTs, the dense representation of SYMPOL corresponds to an equivalent standard DT representation at each point in time, ensuring that the un- derlying model is hard, axis-aligned DT. By enumerating the internal nodes in breadth-fisst order,
4

---

Published as a conference paper at ICLR 2025
2 2.0
22.
00
0.0
5060.13
SGk..)
82 2 1.2
Si 2 0.9
[0.0
00
10
[0.0
1.0
0.0
560).))
56k.m)
-Solu.1)
1L000
[10000)
010.00
(a) Vanilla DT Representation
(b) Dense DT Representation
Figure 2: Standard vs. Dense DT Representation. A comparison between the standard decision tree representation and its dense equivalent, illustrated using an example decision tree of depth 2, with a state space of dimensionality 3 and two possible actions.
we can redefine the indicator function L for a leaf I, resulting in
24
(is)a.T,1)- au Ls((,,
Σ
(3)
where L(s(L,T,1)
II ((,,)) S(s) Tais p(l.j) (1 1--(s)/4a.)) Tai))) y=1
Here, is the index of the internal node preceding leaf node a certain depth j and p indicates whether the left (p 0) or the right branch (p = 1) was taken.
Axis-aligned splitting. Typically, DTs use the Heaviside function for splitting, which is non- differentiable. We use the split function introduced in GradTree to account for reasonable gradients:
S(s|c, r) = [S (e- TTT
(5)
where S(z) Ite represents the logistic function, 1-1 stands for rounding and b denotes the dot product. We further need to ensure that & is a one-hot encoded vector to account for axis-aligned splits. This is achieved by applying a hardmax transformation before calculating S. Both rounding and hardmax operations are non-differentiable and therefore, SYMPOL is not considered as soft or differentiable DT method. Instead, to overcome non-differentiabiiiy, SYMPOL employys a straight- through operator (Bengio al., 2013) during backpropagation. This allows the model to non- differentiable operations in the forward pass while ensuring gradient propagation in the backward directly learn an interpretable DT from policy gradient. This makes
pass. As result, SYMPOL framework-agnostie and facilitates a seamless integration into existing RL frameworks.
Weight decay. In contrast to GradTree, which employs Adam (Kingma & Ba 2014) opti- mizer with stochastic weight averaging (Izmailov et al., 2018), we opted for an Adam optimizer with weight decay (Loshchilov & Hutter 2017). In the context of SYMPOL, weight decay does not serve as regularizer for model complexity, as the interpretation of model parameters differs. We distinguish between three types of parameters: the distributions in the leaves (a), the split index encoding (I), and the split values (I). We do not apply weight decay to the split values because they are independent of magnitude. However, for the split indices and leaves, weight decay enhances exploration during training by penalizing large parameter values. As result, the distribution of the split index selection and class prediction are narrow and have lower variance. This aids dynami- cally adjusting which feature is considered at a split and in altering the predicted leave distribution.
Actor-critic network architecture. Commonly, the actor and critic use a similar network archi- tecture
share the same weights (Schulman et al.) 2017]. While SYMPOL aims for simple and interpretable policy, we do not have the same requirements for the critic. Therefore, we decided to only employ tree-based actor and we can still capture complexity through the value function, without loosing interpretability, as we maintain simple and interpretable policy.
full-complexity NN value function. As result,
Continuous action spaces. Furthermore, we extend the DT policy SYMPOL to environments with continuous action spaces. Therefore, instead of predicting categorical distribution the
5

---

Published as a conference paper at ICLR 2025
classes, we predict the mean of normal distribution at each leaf and utilize an additional variable Fog E TRlA to learn the log of the standard deviation.
4.2 ADDRESSING TRAINING STABILITY
One main challenge when using DTs as a policy is the stability. While a stable training is also desired and often hard to achieve for NN policy, this is even more pronounced for SYMPOL. This is mainly caused by the inherent tree-based architecture. Changing split at the top of the tree can have a severe impact on the whole model, it can completely change the paths taken for certain observations. This is especially relevant in the context of RL, where the data distribution can vary highly between iterations. To mitigate the impact of highly non-stationary training samples, especially at early stages of training, we made two crucial modifications for improved stability.
Motivated by the idea that rollouts of more accurate policies contain increasingly diverse, higher quality samples, we implemented dynamic number of environment steps between training iterations. Let us consider a pendulum as an example. While at early stages of training relatively small sample size facilitates faster learning as the pendulum constantly flips, more optimal policies lead to longer rollouts and therefore more expressive and diverse experiences in the rollout buffer. Similarly, the increasing step counts stabilize the optimization of policy and critic, as the number of experiences for gradient computation grow with agent expertise and capture the diversity within trajectories better. Therefore, our novel collection approach starts with ninit environment steps and expands until fifinal actions are taken before each training iteration. For computational efficiency reasons, instead of increasing the size of the rollout buffer every time step, we introduce a step-wise exponential function. The exponential increase supports exploration in the initial iterations, while maintaining stability later iterations. Hence, we define the number of steps in the environment n2 at time step t as
Exploration stability.
n1e Panit
(1))) 14a
with 1 1 logz
Tlinit
(6)
For our experiments, we define Pinit as a hyperparameter (similar to the static step size for other methods) and set rininal 128 x Pinnt and therefore i which we observed is good default value.
Gradient stability. We also utilize large batch sizes for SYMPOL resulting in less noisy gradients, leading smoother convergence and better stability. In this context, we implement gradient accumulation to virtually increase the batch size further while maintaining memory -eannoor-ffficency.... As reduced noise in the gradients also leads to less exploration in the parameter space, we implement a dynamic batch size, increasing in the same rate as the environment steps between training iterations (Fquation(6) Therefore, we can henefit from exploration and fast convergence early on and increase gradient stability during the training.
5 EVALUATION
We designed our experiments to evaluate whether SYMPOL can learn accurate DT policies without information loss and observe whether the trees learned by SYMPOL are small and interpretable. As mentioned above, we focus on PPO the most prominent actor-critic, on-policy RL algorithm in our evaluation. To support our claim that SYMPOL can be seamlessly integrated into existing on-policy RL frameworks, we additionally provide results using A2C in Appendix A.1
5.1
EXPERIMENTAL SETTINGS
Setup. We implemented SYMPOL in a highly efficient single-file JAX implementation that al- lows a flawless integration with highly optimized training frameworks (Lange [0222] Weng et al.) 2022 Bonnet et al. 2024 I
We evaluated our method on several environments commonly used for benchmarking RI. methods. Specifically, we used the control environments CartPole (CP), Ac- robot (AB), LandrLannerr (L.L., MountainCarContinaa
(MC-C) and Pendulum (PD-C), as well as the MiniGrid (Chevalier- Biiecr--Bisvrttt et al. 2023) environments Empty Empty-Random (E-R), DoorKey (DK),
LavaGap (LG) and SistShift (DS).
OOr implementation is available under https::/githubb.cmm/ss:

---

Published as a conference paper at ICLR 2025
(a) CartPole
(b) LunarLander
(c) Pendulum (cont.)
Figure 3: Selected Training Curves. Shows the training reward of the full-complexity policy (e.g. MLP in the case SA-DT) solid line and the test reward of the interpretable policy as dashed line for three control environments. Additional, more detailed results are in Appendix A.7
Methods. The goal of this evaluation is compare SYMPOL to alternative methods that allow an interpretation of RL policies as a symbolic, axis-aligned DTs. Building on previous work (Silva al., 2020), we employ an MLP and an SDT provide reference to state-of-the-an results and evaluate two methods grounded in the interpretable RI. literature:
State-Action DTs (SA-DT): SA-DTs are the most common method to generate inter- pretable policies post-hoc. Hereby, first train an MLP policy, which is then distilled into a DT as a post-processing step after the training. SA-DT can be considered as a ver- sion of DAGGER (Ross et al.) 1) and therefore simplified version of VIPER (Bastani jet al.) 2018). In a comparative experiment (see Appendix [A.3). we showed that for the case of learning small, interpretable DTs the performance of SA-DT is similar to those of VIPER, which is in-line with results reported e.g. by Kohler et al. (022)..
Discretized Soft DTs (D-SDT): SDTs allow gradient computation by assigning probabili- ties to each node. While SDTs exhibit hierarchical structure, they usually considered as less interpretable, since multiple features are considered a single split and the whole tree is traversed simultaneously (Marton et al., 2024a). Therefore, (Silaa al. (2020) use SDTss as policies which are discretized post-hoc to allow an easy interpretation.
Evaluation procedure. We report the average undiscounted cumulative reward over 5 random trainings with 5 random evaluation episodes each (=25 evaluations for each method). We trained each method for 1iio timesteps. For SYMPOL, SDT and MLP, we optimized the hyperparameters based on the validation reward with optuna (Akiba et al.) 2019) for 60 trials using a predefined grid. For D-SDT we discretized the SDT and for SA-DT, we distilled the MLP with the highest performance. More details on the hyperparameters can be found in Appendix C
5.2 RESULTS
SYMPOL does not exhibit information loss. Ex- isting methods for learning DT policies usually in- volve post-processing to obtain the interpretable model. Therefore, they introduce a mismatch between the op- timized and interpreted policy, which can result in in- formation loss. The main advantage of SYMPOL is the direct optimization of a DT policy, which guaran- tees that there is no information loss between the opti- mized and interpreted policy. To show this, we calcu- lated Cohen's 's D to measure the effect size comparing the validation reward of the trained model with the test reward of the applied, optionally post-processed model (Table T.. We can observe very large effects for SA-DT and D-SDT and only a very small effect for SYMPOL, similar full-complexity models MLP and SDT. This discrepancy can also be observed in the training curves in Figure
Table I: Information Loss. We caleu- lated Cohen's 's D to measure effect size be- tween the validation reward of the trained and the test reward of the applied model. large ef-
Values > 0.8 are considered fect. Detailed results are in Appendix A.2
Cohen's D D
SYMPOL (ours)
-0.019
SA-DT (d=5)
3.449
SA-DT (d=8)
2.527
D-SDT
3.126
MLP
0.306
SDT
0.040

---

Published as a conference paper at ICLR 2025
SYMPOL learns accurate DT poli- cies. We evaluated our approach against existing methods control environments in Table 2 SYM- POL is consistently among the best interpretable models and achieves significantly higher rewards compared to alternative methods for learning DT policies on several environments, espe- cially on LL and PD-C. Further, SYM- POL consistently solves CP and AB and is competitive to full-complexity models on most environments.
Table 2: Control Performance. We report the average undiscounted cumulative test reward over 25 random tri- als. The best interpretable method, and methods not sta- tistically different, marked bold.
CP
AB
LL
MC-C
PD-C
SYMPOL (ours)
500
80
57
94
323
D-SDT
128
-205
-221
-10
-1343
SA-DT (d=5)
446
-97
-197
97
-1251
SA-DT (d=8)
476
75
-150
96
854
MLP
500
72
241
95
191
SDT
500
- 77
-124
4
310
DT policies offer a good inductive bias for categorical environments. While SYMPOL achieves great results in control benchmarks, it may not be an ideal method for environments model- ing physical relationships. As recently also noted by Fuhrer al. (2024). tree- based models are best suited for cate- gorical environments due to their effec- tive use of axis- splits. In our ex- periments on MiniGrid (tubleB).SYN POL achieves comparable superior results to full-complexity models (e.g. SDT
Table 3: MiniGrid Performance. We report the aver- age undiscounted cumulative test reward over 25 random trials. The best interpretable method, and methods not statistically different, are marked bold.
E-R
DK
LG-5
LG-7
DS
SYMPOL (ours)
0.964
0.959
0.951
0.953
0.939
D-SDT
0.662
0.654
0.262
0.381
0.932
SA-DT (d=5)
0.583
0.958
0.951
0.458
0.952
SA-DT (d=8)
0.845
0,961
0,951
0.799
0.954
ML.P
0.963
0.963
0.951
0.760
0.951
0.966
0.959
0.839
0.953
0.954
on LG-7). The performance gap be- tween SA-DT and SYMPOL is smaller in certain MiniGrid enviromments due to less complex en- vironment transition functions and missing randomness, making the distillation easy. Considering more complex environments with randomness or lava like E-R or IG-7, SYMPOL outperforms alternative methods by a substantial margin.
DT policies learned with SYMPOL are small and interpretable. While we trained SYMPOL with depth of and therefore 255 possible nodes, the effective tree size after pruning is sig- nificantly smaller with only 50.5 nodes (internal and leaf combined) on aver- age. This can be attributed to a self- pruning mechanism that is inherently applied by SYMPOL in learning redun- dant paths during the training and there- fore only optimizing relevant parts. Fur- thermore, DTs learned with SYMPOL are smaller than SA-DTs (d=5) with an average of 60.3 nodes and significantly smaller than SA-DTs (d=8) averaging 291.6 nodes. The pruned D-SDT's are significantly smaller with only 16.5, but also have very poor performance, as shown in the previous experiment. An exemplary DT learned by SYMPOL is visualized in Figure Extended results, including a com- parison with SDTs are in Appendix A.4 and A.5
for Brrty
$10
40.0
Figure 4: SYMPOL Policy for MC-C. The main rule en- coded by this DT is that the car should accelerate to the left, if its velocity is negative, and to the right if it is pos- itive. This essentially increases the speed of the car over time, making it possible to reach the goal at the top of the hill. The magnitude of acceleration is mainly determined by the current position, reducing the action cost.
SYMPOL is efficient. In RL, the actor-environment interaction frequently constitutes significant portion of the total runtime. For smaller policies, in particular, the runtime is mainly determined by the time required to execute actions within the environment to obtain the next observation, while the time required to execute the policy itself having a comparatively minimal impact on runtime. Therefore, recent research put much effort into optimizing this interaction through environment
8

---

Published as a conference paper at ICLR 2025
Tum
Figure 6: DistShift. We show the training environment for the agent along with the starting position and goal. The path taken by SYMPOL (see Figure 7 is marked by green arrows and solves the environment.
Figure 7: SYMPOL Policy. This image shows the DT policy SYMPOL. Split nodes visualized as the 3x3 view grid of the agent with one square marking the considered object and position. If the visualized object is present at this position, the true path (left) is taken.
vectorization. The design SYMPOI., in contrast to existing methods for tree-based RI,, allows seamless integration with these highly efficient training frameworks. As a result, the runtime of SYMPOL is almost identical to using an MLP or SDT as policy, averaging less than 30 seconds for Imio timesteps. Detailed results are in Appendix A.7
Ablation study. Section we
SYMPOL
introduced several crucial compo- nents to facilitate an efficient and sta- ble training of SYMPOL. To sup- port the intuitive justifications for our modifications, we performed an abla- tion study (Figure 5) to evaluate the relevance of the individual compo- nents. Our results confirm that each component substantially contributes to the overall performance.
wio separate architecture
dynamic rollout wio batch adjustment
we adamW
0.0
0.2
0.4
0.6
1.0
Normal ized Mean
Figure 5: Ablation Study. We report the mean normalized reward over all control environments (details in Table
CASE STUDY: DETECTING GOAL MISGENERALIZATION
To demonstrate the benefits SYMPOLs enhanced transparency, we present a case study on goal misgeneralization (Di Langosco et al. 2022). Good policy generalization is vital in RL, yet agents often exhibit poor out-of-distribution performance, even with minor environmental changes. Goal misgeneralization is well-researched out-of-distribution robustness failure that occurs when agent learns robust skills during training but follows unintended goals. This happens when the agent's behavioral objective diverges from the intended objective, leading high rewards during training but poor generalization during testing. For instance, NNs were shown to systematically misgeneralize Atari environments (Farebrother et al. 2018 Delfosse et al. 2024a).
To demonstrate that SYMPOL can help in detecting misaligned behavior, let us consider the Dist- Shift environment from MiniGrid, shown in Figure [ The environment is designed test for misgeneralization Schevllie-Boiiverr et al. 2023). as the goal is repeatedly placed in the top right corner and the lava remains the same position. We can formulate the intended behavior accord- ing to the task description avoiding the lava and reaching specific goal location. SYMPOL, similar other methods, solved the task consistently. The advantage SYMPOL the tree-based structure, which is easily interpretable. When inspecting the SYMPOL policy (Figure 7 we can immediately observe that the agent has not captured the actual task correctly. Essentially. it has only leamed to keep an empty space the left of the agent (which translates into following the wall) and not to step into lava (but not to get around it). While this sufficient to solve this exact environment, it is evident, that the agent has not generalized to the overall goal.

---

actually Published as a conference paper at ICLR 2025
braed
Figure 8: DistShift with Domain Random- ization. This is a modified version of Dist- Shift, with goal and lava at random positions, SYMPOL (Figure 7) visualized as the green line, is not able to solve the randomized en- vironment. Training SYMPOL with domain randomization (Figure 9). visualized as the red line, is able to solve the environment.
Figure 9: SYMPOL Policy with Domain Ran- domization. The SYMPOL policy (Figure 9 re- trained with domain randomization. The agent now has learned to avoid lava and walls, as well identifying and walk into the goal.
In order to test for misgeneralization, we created test environments in which the agent has to reach random goal placed with lava placed varying locations. As already identified based on the interpretable policy, we can observe in Figure g that the agent gets stuck when the goal or lava positions change. Alternative non-interpretable policies exhibit the same behavior, which might remain unnoticed due to the black-box nature. Instead of simply looking at the learned policy with SYMPOL, alternative methods would require using external methods or designing complex test cases to detect such misbehavior. Alternative methods generate DT policies like SA-DT also provide an interpretable policy, but as already shown during our experiments, frequently come with severe information loss. Due to this information loss, cannot ensure that interpreting the policy, which is guaranteed using SYMPOL.
Based on these insights, retrained SYMPOL with domain randomization. The resulting pol- icy (see Figure [ now solves the randomized environments (see Figure 8). still maintaining in- terpretability. In line with our results, Delfosse al. (2024c) showed that using interpretable DT policies can help to mitigate goal misgeneralization which showcases the potential benefit of using interpretable RI. policies ensure good generalization.
CONCLUSION AND FUTURE WORK
In this paper, we introduced SYMPOL, a novel method for tree-based RL. SYMPOL can be seam- lessly integrated into existing on-policy RI. frameworks, where the DT policy is directly optimized on-policy while maintaining interpretability. This direct optimization guarantees that the explanation exactly matches the policy learned during training, avoiding the information loss often encountered with existing methods that rely on post-processing to obtain an interpretable policy. Furthermore, the performance of interpretable DT policies learned by SYMPOL is significantly higher compared to existing methods, particularly in environments involving more complex environment transition functions randomness. We believe that SYMPOL represents significant step towards bridging the gap between the performance of on-policy RI and the interpretability and transparency of sym- bolic approaches, paving the way for the widespread adoption of trustworthy and explainable AI systems in safety-critical and high-stakes domains.
While we focused on an actor-critic on-policy RL method, the flexibility of SYMPOL allows an integration into arbitrary policy optimization RL frameworks including off-policy methods like Soft Actor-Crite (SAC) (Haarnoja et al. 2018) or Advantage Weighted Regression (AWR) (Peng et al. 2019), which be explored in future work. Also, it would be interesting to evaluate more complex, forest-like tree structure as a perfommanceineerrrttbbllll interpretabiliity trade-off, similar to Mo/Maton et al. 2024b), especially based the promising results of Fuhrer et al. (2024) for tree-hased RI..
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was supported in part by the German Federal Ministry for Economic Affairs and Climate Action of Germany (BMWK), and in part by the German Federal Ministry of Education and Research (BMBF).
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, pp. 2623-2631, 2019.
Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. Advances in Neural Information Processing Systems, 31, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: /30iv3300..330 34322 2013.
Clément Bonnet, Daniel Luo, Donal Byrne, Shikha Surana, Sasha Abramowitz, Paul Duckworth, Vincent Coyette, Laurence I. Midgley, Elshadai Tegegn, Tristan Kalloniatis, Omayma Mahjoub, Matthew Macfarlane, Andries P. Smit, Nathan Grinsztajn, Raphael Boige, Cemlyn N. Waters, Mohamed Mimouni, UIrich A. Mbou Sob, Ruan de Kock, Siddarth Singh, Daniel Furelos- Blanco, Victor Le, Arnu Pretorius, and Alexandre Laterre. Jumanji: a diverse suite of scalable reinforcement learning environments in jax, 2024. URL http://axivvvv//. org hbspg//bb/23066 09884]
Yushi Cao, Zhiming Li, Tianpei Yang, Hao Zhang, Yan Zheng, Yi Li, Jianye Hao, and Yang Liu. Galois: boosting deep reinforcement learning via generalizable logic synthesis. Advances in Neural Information Processing Systems, 35:19930-19943 2022.
Chun-Hao Chang, Rich Caruana, and Anna Goldenberg. Node-gam: Neural generalized additive model for interpretable deep learning. arXiv preprint arXiv:2106.01611 2021.
Maxime Chevelier-Boisvertt, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306.13831, 2023.
Youri Coppens, Kyriakos Efthymiadis, Tom Lenaerts, Ann Nowé, Tim Miller, Rosina Weber, and Daniele Magazzeni. Distilling deep reinforcement learning policies in soft decision trees. In Proceedings the
the IJCAI 2019 workshop on explainable artificial intelligence, pp. 1-6, 2019.
Vinícius G Costa, Jorge Pérez-Aras Aracil, Sancho Salcedo-Sanz, and Carlos Pedreira. Evolving inter- pretable decision trees for reinforcement learning. Artificial Intelligence, 327: 104:57 2024.
Leonardo L Custode and Giovanni Iacca. Evolutionary learning of interpretable decision trees. IEEE Access, 11:6169-611 6184, 2023.
Quentin Delfosse, Jannis Blüml, Bjarne Gregori, and Kristian Kersting. Hackatari: Atari learning environments for robust and continual reinforcement learning. arXiv preprint 2XXx:2406000997,, 2024a.
Quentin Delfosse, Hikaru Shindo, Devendra Dhami, and Kristian Kersting, Interpretable and ex- plainable logical policies via neurally guided symbolic abstraction. Advances Neural Informa- fion Processing Systems, 36, 2024b.
Quentin Delfosse, Sebastian Sziwiertnia, Mark Rothermel, Wolfgang Stammer, and Kristian Ker- sting. Interpretable concept bottlenecks to align reinforcement learning agents. arXiv preprint arXiv:2401.05821, 2024c.
Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal mis- generalization in deep reinforcement learning. In International Conference on Machine Learning, pp. 12004-12019, PMLR, 2022.
11

---

Published as a conference paper at ICLR 2025
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint urXiv:1810.00122. 2018.
Gregory Farquhar, Tim Rocktäschel, Maximilian Igl. and Shimon Whiteson. Treeqn and atreec: Differentiable tree-structured models for deep reinforcement learning. arXiv preprint arXiv:1710.1 2017.
Benjamin Fuhrer, Chen Tessler, and Gal Dalal. Gradient boosting reinforcement learning. arXiv preprint arXiv:2407.062550 2024.
Jiaming Guo, Rui Zhang. Shaohui Peng, Qi Yi, Xing Hu, Ruizhi Chen, Zidong Du, Ling Li, Qi Guo, Yunji Chen, et al. Efficient symbolic policy learning with differentiable symbolic expression. Advances in Neural Information Processing Systems, 36, 2024.
Ujjwal Das Gupta, Erik Talvitie, and Michael Bowling. Policy tree: Adaptive representation for policy gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. PMLR, 2018
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
Aman Jhunjhunwala. Jaeyoung Lee, Sean Sedwards, Vahdat Abdelzad, and Krzysztof Czarnecki. Improved policy extraction via online v-value distillation. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
Zhengyao Jiang and Shan Luo. Neural logic reinforcement learning. In International conference on machine learning, pp. 3110-3119, PMLR, 2019.
Akansha Kalra and Daniel S Brown, Interpretable reward learning via differentiable decision trees. In NeurlPS ML Safety Workshop, 2022.
Akansha Kalra and Daniel S Brown. Can differentiable decision trees learn interpretable reward functions? arXiv preprini arXiv:2306. 2023.
Alexare Kamienny et al. Symbolic-model--asse reinforcement learning. In NeurIPS 2022 Al for Science: Progress and Promises, 2022.
Kentaro Kanamori, Takuya Takagi, Ken Kobayashi, and Yuichi Ike. Counterfactual explanation trees: Transparent and consistent actionable recourse with decision trees. In International Con-
ference on Artificial Intelligence and Statistics, pp. 1846-1870. PMLR, 2022.
Eoin M. Kenny, Mycal Tucker, and Julie Shah. Towards interpretable deep reinforcement learn- ing with human-friendly prototypes. In International Conference Learning Representations (ICLR), 2023. URL https: wtwpeerrerieeeeeeeeoouummmmeemee
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69988 6980, 2014.
Hector Kohler, Quentin Delfosse, Riad Akrour, Kristian Kersting, and Philippe Preux. Inter- pretable and editable programmatic tree policies for reinforcement learning. arXiv preprint arXiv:2405. 14956, 2024.
Mikel Landajuela, Brenden Petersen, Sookyung Kim, Claudio P Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel M Faissol. Discovering symbolic policies with deep rein- forcement learning. In Proceedings of the 38th International Conference on Machine Learning, pp. 5979-5989. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Robert Tjarko Lange. gymnax: A JAX-based reinforcement learning environment library, 2022.
URL http::ttttttiiiie http://gggtthu....oo///eeeartttt Cow.eeoaaaatttaaaaggeeeeeeeeeeee
qyymax
Zhao-Hua Li, Yang Yu, Yingfeng Chen, Ke Chen, Zhipeng Hu, and Changjie Fan. Neural-to-tree policy distillation with policy improvement criterion. arXiv preprint arXiv:2108.0668 2021.
Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li. Toward interpretable deep reinforce- ment learning with linear model u-trees. Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part II 18, pp. 414-429. Springer, 2019.
Xiao Liu, Shuyang Liu, Bo An, Yang Gao, Shangdong Yang, and Wenbin Li. Effective interpretable policy distillation via critical experience point identification. IEEE Intelligent Systems, 38(5): 28-36, 2023.
Zichuan Liu, Yuanyang Zhu, Zhi Wang, Yang Gao, and Chunlin Chen. Mixits: Toward in- terpretable multi-agent reinforcement learning via mixing recurrent soft decision trees. arXiv preprint M.XXv:2209.07225. 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101, 2017.
Lirui Luo, Guoxi Zhang, Hongming Xu, Yaodong Yang, Cong Fang, and Qing Li. End-to-end neuro-symbolic reinforcement learning with textual explanations. In Forty-first International Conference on Machine Learning.
Lirui Luo et al. End-to-end neuro-symbolic reinforcement learning with textual explanations. arXiv preprint arXiv:2403. 2024.
Sascha Marton, Stefan Lüdtke, Christian Bartelt, and Heiner Stuckenschmidt. Gradtree: Learning axis-aligned decision trees with gradient descent. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 14323 14331, 2024a.
Sascha Marton, Stefan Lüdtke, Christian Bartelt, and Heiner Stuckenschmidt. Grande: Gradient- based decision tree ensembles for tabular data. In The Twelfth International Conference on Learn- ing Representations, 2024b.
Stephanie Milani, Zhicheng Zhang, Nicholay Topin, Zheyuan Ryan Shi, Charles Kamhoua, Evan- gelos Papalexakis, and Fei Fang. Maviper: Learning decision tree policies for interpretable multi-agent reinforcement learning. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 251-266. Springer, 2022.
Joosung Min and Lloyd T Elliott. Q-learning with online random forests. arXiv preprint arXiv:2204.03771, 2022.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning, 2016. htthhttp:://arrtt wttp://wwww.////////o///////
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 190.00779, 2019.
Svetlin Penkov and Subramanian Ramamoorthy, Learning programmatically structured representa- tions with perceptor gradients. arXiv preprint wwXiv:1905.00956 2019.
Ben Peters, Vlad Niculae, and André FT Martins. Sparse sequence-to-sequeen models. arXiv preprint 1rXXiv1905.00502 2019.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv: /9099063/2933 2019.
Stéphane Ross, Geoffrey Gordon, and Andrew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the 14th International Con- ference on Artificial Intelligence and Statistics (AISTATS), pp. 637-635. JMLR Workshop and Conference Proceedings, 2011.
13

---

Published as a conference paper at ICLR 2025
Aaron M Roth, Nicholay Topin, Pooyan Jamshidi, and Manuela Veloso. Conservative q- improvement: Reinforcement learning for an interpretable decision-tree policy. arXiv preprint arXiv:1907.01180, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint orXiv:1777003444 2017.
Andrew Silva and Matthew Gombolay. Encoding human domain knowledge forcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 5042-5050, 2021.
rein-
Andrew Silva, Matthew Gombolay, Taylor Killian, Ivan Jimenez, and Sung-Hyur Son. Optimiza- tion methods for interpretable differentiable decision trees applied to reinforcement learning. In International conference on artificial intelligence and statistics, pp. 1855-18888888 1865. PMLR, 2020.
Pradyumna Tambwekar, Andrew Silva, Nakul Gopalan, and Matthew Gombolay. Natural lan- guage specification reinforcement learning policies through differentiable decision trees. IEEE Robotics and Automation Letters, 8(6):3621-362) 2023.
Nicholay Topin, Stephanie Milani, Fei Fang, and Manuela Veloso. Iterative bounding mdps: Learn- ing interpretable policies via non-interpretable methods. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9923-9931, 2021.
Mark Towers, Ariel Kwiatkowski, Jordan Terry John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407. 17032, 2024.
Dweep Trivedi, Jesse Zhang. Shao-Hua Sun, and Joseph J Lim. Learning to synthesize programs as interpretable and generalizable policies. Advances Neural Information Processing Systems, 34:25146-25163 2021.
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pp. 5045-5054. PMLR, 2018.
Daniël Vos and Sicco Verwer. Optimal decision tree policies for markov decision processes. arXiv preprint 1311122301.13185, 2023.
Daniel Vos and Sicco Verwer. Optimizing interpretable decision tree policies for reinforcement learning, arXiv preprint arXiv:2408. /632, 2024. URL /Bt./t/pss.////vv...o//aa/////// 11632
Maxime Wabartha and Joelle Pineau. Piecewise linear parametrization of policies: Towards in- terpretable deep reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. htllhttps:/// httpss:/openzeview.
Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makovicchuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and Shuicheng Yan. En- vPool: A highly parallel reinforcement learning environment execution engine. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 22409-22421. Curran Associates, Inc., 2022. /ttpa///cccaaapoeeae/e//////////
URL. httpps//pproceeeingg.eeerr neur ips cc/ fi les pppps///////////1leeeee
Beaadlee9ocaaa6sttee077222222222 Datasets. and Benchmarks.
pdf
Zhuorui Ye, Stephanie Milani, Fei Fang, and Geoff Gordon. Concept-based interpretable rein- forcement learning with limited to no human labels. In Workshop on Interpretable Policies in Reinforcement Learning @ RLC-2024, 2024.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RESULTS
In this section, we present additional results to support the claims made in the main paper, along with extended results for the summarizing tables. We focus on the control environments because they offer diverse suite of benchmarks that cover different tasks and include both continuous and discrete action spaces. We chose not to include the MiniGrid environments here because their inclu- sion could distort the results, particularly the averages calculated in the main paper, as all MiniGrid environments involve similar tasks and feature discrete action and observation space. The primary reason for including MiniGrid in the main paper is to provide additional experimental results that confirm the robustness and applicability of our method across different domains, as well as to high- light that tree-based methods offer beneficial inductive bias for these categorical environments.
A.1 EVALUATION WITH ALTERNATIVE RL ALGORITHMS: ADVANTAGE ACTOR CRITIC (A2C)
To support our claim that SYMPOL can integrated into arbitrary on-policy frameworks, we pro- vide results A2C in the following. The reported results use optimized hyperparameters for each method. general, A2C is considered as less stable compared to PPO, which is an additional challenge for SYMPOL, training stability is especially crucial for DT policies. As A2C does not update the policy in minibatches over multiple epochs, we did not include dynamic batch size here, but update SYMPOL with single update over the rollout to stay consistent with the A2C algorithm.
Table 4: A2C Control Performance. We report the average undiscounted cumulative test reward over 25 random trials. The best interpretable method, and methods not statistically different, are marked bold.
CP
AB
LL
MC-C
PD-C
SYMPOL (ours)
500
84
85
58
502
D-SDT
-396
-0
-1137
SA-DT (d=5)
295
-102
-348
-1467
SA-DT (d=8)
223
99
-367
2
-1526
MLP
500
78
208
202
SDT
500
85
-159
-011
Table 5: A2C Control Performance Comparison. We report the average undiscounted cumulative test reward over 25 random trials, comparing A2C with PPO using optimized hyperparameters. A number is marked bold if the performance achieved with the underlying RI. algorithm (PPO A2C) is significantly better or not statistically different from the best result.
MLP
SDT
SYMPOL (ours)
SA-DT (d=5)
SA-DT (d=8)
D-SDT
A2C
PPO
A2C
PPO
A2C
PPO
A2C
PPO
A2C
PPO
A2C
PPO
CP
500
500
500
500
500
500
295
446
223
476
11
128
AB
-78
-72
-85
-77
-84
-80
-102
-97
-99
-75
-427
-205
LL
208
241
-159
-124
-85
-57
-348
-197
-367
-150
-396
-221
MC-C
0
95
-4
58
94
97
-2
96
0
-10
PD-C
-202
-191
-201
-310
-502
-323
-1467
-1251
-1526
-854
-1137
-1343
We compared the performance of all methods using A2C on control environments in Table and additionally provided a direct comparison between PPO and A2C in Table When using A2C, SYMPOL consistently outperforms other interpretable models. The performance gap becomes even more pronounced when using A2C instead of PPO, as SYMPOL achieves substantially higher per- formance than all other interpretable models in each environment. On MC-C, SYMPOL is the only method that achieves a positive reward, whereas even full-complexity models were unable to solve the task. This can be attributed the lower training stability of A2C compared to PPO. This could also explain the poor results of distillation methods, as the policy learned by full-complexity models, even when achieving high test reward, is potentially less consolidated, making it harder to distill.
15

---

Published as a conference paper at ICLR 2025
However, to confirm this assumption, further experiments would be required. Based these results, we can confirm that SYMPOL can seamless be integrated into other RL algorithms, demonstrating the high flexibility of our proposed method. Additionally, our method can benefit from advances in RI, as it can be seamlessly integrated into novel frameworks.
A.2 INFORMATION LOss
We provide detailed results on the information loss which can result as a consequence of discretiza- tion (for D-SDT) or distillation (for SA-DT). In Table [6] we report the validation reward of the trained model along with the test reward of the discretized model. We can clearly observe that there are major differences for SA-DT and D-SDT on several datasets, indicating information loss. In Table 7 we report Cohen's D to measure the effect size comparing the validation reward of the trained model with the test reward of the applied, optionally post-processed model. Again, we can clearly see large effects for SA-DT and D-SDT on several datasets, especially for PD-C and LL, but also CP. Furthermore, the training curves in Figure 14 visually show the information loss during the training.
Table 6: Information Loss (Comparison). We report the validation reward of the trained model and the test reward of the applied model.
MLP
SDT
SYMPOL (ours)
SA-DT (d=5)
SA-DT (d=8)
D-SDT
valid
test
valid
test
valid
test
valid
test
valid
test
valid
test
CP
500
500
500
500
500
500
446
500
500
128
AB
-71
-72
-77
-79
-80
-71
-97
-71
-75
-89
-205
LL
256
-124
-57
256
256
-150
-91
-221
MC-C
95
95
87
94
95
97
95
96
-10
PD-C
-169
-191
-295
-310
-305
-323
-169
-1251
-169
-854
-295
-1343
Table 7: Information Loss (Cohen's D). We calculated Cohen's D to measure effect size between the validation reward of the trained model and the test reward of the applied model. Typically, values > 0.5 are considered a medium and values > 0.8 large effect. positive effeets that are least medium are marked as bold.
MLP
SDT
SYMPOL (ours)
SA-DT (d=5)
SA-DT (d=8)
D-SDT
CP
0.000
0.000
0.000
0.632
1.214
4.075
AB
0.035
-0.630
0.104
0.728
0.338
0.982
LL
0.341
0.750
0.370
4.776
8.155
2.254
MC-C
-0,042
-0.002
-1.035
-2,011
-1.172
0.745
PD-C
1.195
0.081
0.468
13.120
4.101
7.573
Mean
0.306
0.040
-0.019
3.449
2.527
3.126
A.3 COMPARISON OF SYMPOL, SA-DT (DAGGER) AND VIPER (Q-DAGGER)
In this section, we provide a direct comparison of SYMPOL with SA-DT and VIPER for control environments (see Table 8 and MiniGrid environments (see Table 9) SA-DT (Silva al. 2020 can be considered as a version of DAGGER (Ross et al.) 2011) and is conceptually similar to VIPER (Q-DAGGER) (Bastami al. 2018) which improves data collection including additional weighting. Our results remain consistent with our original claims, demonstrating that SYMPOL outperforms alternative approaches. This is also in-line with the results reported by Kohler et al. (2024), where the authors show, that the sampling in VIPER does not yield a better performance compared to DAGGER/SA-DT for interpretable DTs. The results reported in the original VIPER paper (Bastani et al. 2018) stating achieve perfect reward for CP are on different version of the environment (CartPole-vO) with only 200 opposed to 500 time steps and less randomness (CartPole-v) 1)), making the underlying task casier. Also, want to note, that the reported results in-line with related work, reporting comparable or worse results than ours. For instance, Vos & Verwer (2024) report mean reward of only 367 for VIPER C Also, Kenny al. 2023) showcase poor
16

---

Published as a conference paper at ICLR 2025
Table 8: Control Performance. We report the average undiscounted cumulative test reward over 25 random trials. The best interpretable method, and methods not statistically different, are marked bold. Please note that VIPER cannot be applied to continuous environments.
CP
AB
LL
MC-C
PD-C
SYMPOL (ours)
500
80
57
94
323
D-SDT
128
-205
-221
-10
-134
SA-DT (d=5)
-97
-197
97
-1251
SA-DT (d=8)
476
75
-150
96
854
VIPER (d=5)
457
77
-200
VIPER (d=8)
480
75
-169
MLP
500
72
241
95
191
SDT
500
77
-124
310
Table 9: MiniGrid Performance. We report the average undiscounted cumulative test reward over 25 random trials. The best interpretable method, and methods not statistically different, are marked bold.
E-R
DK
LG-5
LG-7
DS
SYMPOL (ours)
0.964
0.959
0.951
0.953
0,939
D-SDT
0.662
0.654
0.262
0.381
0.932
SA-DT (d=5)
0.583
0.958
0.951
0.458
0.952
SA-DT (d=8)
0.845
0.961
0.951
0.799
0.954
VIPER (d=5)
0.651
0.958
0.948
0,456
0.954
VIPER (d=8)
0.845
0.963
0,948
0.801
0,954
MLP
0.963
0.963
0.951
0.760
0,951
SDT
0.966
0.959
0.839
0.953
0.954
performance of VIPER in general and specifically for LL the performance is worse than what we reported. Our findings align with these, suggesting that differences in performance may reflect randomness and missing generalizability in the evaluation.
A.4
TREE SIZE
We report the average tree sizes over 25 trials for each environment. The DTs for SYMPOL and D-SDT are automatically pruned by removing redundant paths. There are mainly two identifiers, making path redundant:
The split threshold of split is outside the range specified by the environment. For instance, if æ E [0.0, 1.0] the decision 2 < -0.1 will always false as -0.1 < 0.0.
A decision at a higher level of the tree already predefines the current decision. For instance, if the split at the root node æ1 < 0.5 and the subsequent node following the true path 0.6 we know that this node will always be evaluated to true as 0.5 < 0.6.
We excluded the MiniGrid environments here, as they require a more sophisticated, automated prun- ing as there exist more requirements making path redundant. For instance, if for the decision whether there a wall in front of the agent is true, the decision for all other objects at the same position has to be always false.
17

---

Published as a conference paper at ICLR 2025
Table 10: Tree Size. We report the average size of the learned DT for each environment.
SYMPOL (ours)
D-SDT
SA-DT (d=5)
SA-DT (d=8)
CP
39.4
14.2
61.8
315.0
AB
78.6
17.0
56.5
173.0
LL
55.0
19.8
59.8
270.2
MC-C
23.4
3.0
61.0
311.8
PD-C
56.2
28.6
62.2
388.2
Mean
50.5
16.5
60.3
291.6
A.5 INTERPRETABILITY COMPARISON BETWEEN HARD, AXIS-ALIGNED AND SOFT DT'
In the main paper, we showed visualization of a hard, axis-aligned DT learned by SYMPOL on the MC-C environment. While this was comparatively small tree, we provide another example of comparatively large tree with 59 nodes in Figure TO While the tree is comparatively large, we can observe that the main logic is contained in the nodes higher levels, focusing on the pole angle and the pole angular velocity. The less important features are in the lower levels where splits are often mage on the cart position, which not required to solve the task perfectly. This also highlights potential for advanced post-hoc pruning methods to increase interpretability and potentially even generalization. When comparing the hard, axis-aligned decision trees (DTs) (Figure IO and Fig- 4re learned by SYMPOL with corresponding soft decision trees (SDTs) learned by existing direct optimization methods (Figure and Figure 12). the superiority of axis-aligned splits over oblique, multidimenssioal boundaries becomes evident.
The DTs learned by SYMPOL are substantially more interpretable, both at the level of individual splits and the overall tree structure. For example, when examining the node of the CartPole task, the standard DT (Figure 10p makes a straightforward comparison, "sy(Pole Velocity) v -0.800", whereas the corresponding SDT (Figure IT] expresses the decision as "o(-1114sp +.. - 0.99)", which is significantly more complex and harder to interpret. This disparity becomes even more pronounced when considering complete paths or the entire tree. In SYMPOL's DTs, decisions at nodes are binary (yes/no), whereas SDTs employ probabilistic routing.
0.30s; +
Probabilistic routing introduces two key disadvantages interpretability compared axis-eligned DTs: (1) the need to consider multiple paths simultaneously, and (2) the inability to directly interpret the leaf outputs, as they are weighted by path probabilities. We also want to note that in order to allow a visualization of the SDT, we reduced the tree size to only 4, while the tree size used during the evaluation was 7.
A.6 ABLATION STUDY
Our ablation study was designed to support our intuitive justifications for the modifications made to the RI framework and our method. Therefore, we disabled individual components of our method and evaluated the performance without the specific component. This includes the following modifi- cations introduced in Section 4:
1. w/o separate architecture: Instead of using separate architectures for actor and critic, we use the same architecture and hyperparameters for the actor and critic.
2. w/o dynamic rollout: exponential rate during training to increase stability while maintaining exploration early on. Here we used a standard, static rollout buffer.
We proposed dynamic rollout buffer that increases with stepwise
3. w/o batch size adjustment: Similar the dynamic rollout buffer, we proposed using dynamic batch size to increase gradient stability in later stages of the training. Here, we used standard, static batch size.
4. w/o adamW:
We introduced an Adam optimizer with weight decay to SYMPOL to support the adjustment of the features to split on and the class predicted. Here, we use a standard Adam optimizer without weight decay.
18

---

Published as a conference paper at ICLR 2025
Figure 10: Exemplary large CartPole DT. This figure visualizes larger tree with 59 nodes learned by SYMPOL on the CartPole environment.
19

---

Published as a conference paper at ICLR 2025
1.0- C3n
Figure 11: Exemplary SDT for CartPole. This figure visualizes soft differentiable DT learned on CartPole. The tree involves oblique decisions involving multiple features at each split. Addi- tionally, there no hard decision on which path is selected, but multiple paths are taken with an associated probability. The final prediction is obtained by weighting the leaf outputs with the corre- sponding path probability.
20

---

Published as a conference paper at ICLR 2025
Figure 12: Exemplary SDT for MountainCar. This figure visualizes a soft / differentiable DT learned on MountainCar. The tree involves oblique decisions involving multiple features at each split. Additionally, there is no hard decision on which path selected, but multiple paths are taken with associated probability. The final prediction is obtained by weighting the leaf outputs with the corresponding path probability.
21

---

Published as a conference paper at ICLR 2025
Detailed results for each of the control datasets reported in Table 11 The results clearly confirm our intuitive justifications, as each adjustment has a crucial impact on the performance SYMPOL.
Table 11: Ablation Study. We report the average test performance over total of 23 random trials. This normalized performance consists in normalizing each reward between and I via an affine renormalization between the top- and worse-performing models. Instead of the worse-performing model, we use the 20% test reward quantile to account for outliers.
Agent Type
CP
MC-C
PD-C
Normalized Mean (t)
SYMPOL
500.0
79.9
57.4
94.3
323.3
0.988
separate architecture
135.6
-196.4
-276.8
-55.44
-1219.4
0,080
w/o dynamic rollout
456.1
92.0
-178.2
-144.1
433.4
0.598
w/o batch size adjustment
498.8
81.7
-320.7
-1818.8
-434.4
0.372
w/o adamW
416.1
78.3
97.3
0.0
393.5
0.865
A.7
RUNTIME AND TRAINING CURVES
The experiments were conducted single NVIDIA RTX A6000. The environments for CP, AB, MC-C and PD-C were vectorized (Lange 2022) and therefore the training is highly efficient, taking only 30 seconds for Imio timesteps on average (excluding the sequential evaluation which cannot be vectorized). The remaining environments are not vectorized, and we used the standard Gymna- sium (Towers et al. 2024) implementation. In Table [2 can clearly see the impact of environment vectorization, as the runtime for LL., which is not vectorized, is more than 10 times higher with over 400 seconds.
Table 12: Runtime. We report the average runtime 25 trials. One trial spans Imio timesteps for each environment. We excluded LL from the mean runtime calculation, as this is the only non- vectorized environment. To provide fair comparison of different methods, we aligned the step and batch size.
SYMPOL (ours)
SDT
MLP
CP
28.8
23.9
25.2
AB
35.5
37.7
33.8
MC-C
23.4
19.4
18.4
PD-C
28.7
28.2
18.5
Mean
29.1
27.3
24.0
LL
402.3
394.0
405.6
In addition to the training times, we report detailed training curves for each method. Figure 13 compares the training reward and the test reward of SYMPOL with the full-complexity models MLP and SDT. SYMPOL shows similar convergence compared to full-complexity models on environments. For AB, SYMPOL converges even faster than an MLP which can be attributed to the dynamic rollout buffer and batch size. For MC-C we can see that the training of SYMPOL. is very unstable at the beginning. We believe that this can be attributed to the sparse reward function of this certain environment and the fact that severe drop in the reward. Combined with the small rollout buffer and batch size carly in the training of SYMPOL, this can result in unstable training. However, later on, which again confirms the design of our dynamic buffer size increasing over time.
result, minor changes in the policy can result in
that the training stabilizes
Furthermore, we provide pairwise comparison of SYMPOL with SA-DT and D-SDT in Figure 14 Here, we can again observe the severe information loss for D-SDT and SA-DT by comparing the training curve with the test reward.
22

---

Published as a conference paper at ICLR 2025
0100
(a) CP
(b) AB
(L)
(d) MC-C
PD-C
Figure 13: Training Curves (Full-Complexity). Shows the training reward solid line and the test reward as dashed line for SYMPOL (blue), MLP (orange) and SDT (green).
23

---

Published as a conference paper at ICLR 2025
(a) CP SYMPOL vs. SA-DT (d=5)
(b) CP SYMPOL SA-DT (d=8)
CP SYMPOL D-SDT
(d) AB SYMPOL vs. SA-DT (d=5)
(c) AB SYMPOL vs. SA-DT (d=8)
(f) AB SYMPOL vs. D-SDT
(g) LL SYMPOL vs. SA-DT (d=5)
(h) LL SYMPOL SA-DT (d=8)
(i) LL S.MPOYM vs. D-SDT
(i) MC-C SYMPOL SA-DT (k) MC-C SYMPOL vs. SA-DT
(l) MC-C vs. D-SDT
(d=5)
(d=8)
Gatal Secrrd
(m) PC-C SYMPOL SA-DT (n) PC-C SYMPOL vs. SA-DT (d=5)
(0) PC-C D-SDT
(d=8)
Figure 14: Training Curves. Shows the training reward solid line and the test reward as dashed line for SYMPOL (blue), SA-DT-5 (orange) SA-DT-8 (green) and D-SDT (red). Thereby, the test reward is calculated with the diserretizedddistilleed policy for SA-DT and D-SDT. For several datasets,
we can again observe the severe information loss introduced with the post- post-processing (e.g. for PD-C and LL).
B MINIGRID
We used the MiniGrid implementation from (Chevalier-Boisvent Boisvert et al., al.) For each environment, limited to observations and the actions to the available ones according to the documentation. Furthermore, we decided to use a view size of 3 to allow a good visualization the results. In the following, provide more examples for MiniGrid Use-Case, along with detailed visual-
24

---

Published as a conference paper at ICLR 2025
izations. In the following, visualized the SYMPOL agent sequentially acting in the environment as one image for one step from left to right and top to bottom. Figure 15 shows how SYMPOL (see image in the main paper tree function (obs) defined below) solves the environment. Fig- ure and Figure 18 show the same agent failing on the environment with domain randomization, proving that the agent did not generalize, tree-based policy. Retraining the agent with domain randomization (see image in the main paper or tree_funct cettiinnree www.airraredd.. (obs) defined below), SYMPOL is able solve the environment (see Figure 17 and Figure 19), maintaining interpretability.
could already observe by inspecting the symbolic,
B.1 VISUALIZATIONS ENVIRONMENT
Figure 15: DistShift SYMPOL. This figure visualizes the path taken by the SYMPOL. agent trained on the basic DistShift environment (see image in the main paper or tree srreeffucctie ion (obs) ) de- fined below) from left to right and top to bottom. The agent follows the wall and reaches the goal at the top right corner.
25

---

Published as a conference paper at ICLR 2025
Figure 16: DistShift (Domain Randomization) SYMPOL Example This figure visualizes the path taken by the SYMPOL agent trained on the basic DistShift environment (see image in the main paper or tree fene.functii (obs) ) defined below) from left to right and top to bottom. The agent follows the wall gets stuck by the lava.
Figure 17: DistShift (Domain Randomization) SYMPOL (retrained) Example 1. This figure
visualizes the path taken by the SYMPOL agent trained on the randomized DistShift environment (see image in the main paper or tree funct i wwwoooterrrtaaeeeooeoeeeeeee retrainedd (cbs) defined below) from left right and top to bottom. The agent avoids the lava, identifies the goal and walks into the goal.
26

---

Published as a conference paper at ICLR 2025
Figure 18: DistShift (Domain Randomization) SYMPOL Example 2. This figure visualizes the path taken by the SYMPOL agent trained on the basic DistShift environment (see image in the main paper or reee_ fese_functti on (obs) defined below) from left to right and top to bottom. The agent follows the wall until there is empty space the left. Instead of empty space there is the goal, but instead walking into the goal, the agent surpasses it and again gets stuck at the lava.
Figure 19: DistShift (Domain Randomization) SYMPOL (retrained) Example 2. This figure visualizes the path taken by the SYMPOL agent trained on the randomized DistShift environment (see image the main paper or tree funct reriiiottrrrrreeeeee ret ned (abs) defined below) from left to right and top to bottom. The agent avoids the lava, identifies the goal, and walks into the goal.
27

---

Published as a conference paper at ICLR 2025
B.2 SYMPOL ALGORIHHMIC PRESENTATION
def tree_function (obs):
if obs to front and one to left] is emmpyy if cbs(field if obs [field one to left] is 'empty':
front] is 'lava'
action 'turn right'
else:
action 'turn left'
else:
action move forward'
10
else:
action 'turn left'
12
return action
def
if (fiifi one left] "goal'; action 'turn left'
else:
if obs(field one to right] is 'goal': action 'turn right"
eise:
is obslfield two to trent] is 'wall':
if obs[field one to front and one to right] is "goal': if obs [field cne to right] is (lava' action 'turn left'
12
else:
13
action move forward'
else:
if obs (field
front] is 'goal'; action 'ove forward'
16
17
else:
18
action turn left'
else:
if obs [field one front] is (lava': action tturn right'
21
22
else:
action 'evee forward'
return action
C METHODS AND HYPERPARAMETERS
The main methods we compared SYMPOL against are behavioral cloning state-action DTs (SA-DT) and discretized soft decision trees (D-SDT). In addition the information given in the paper, want to provide some more detailed results of the implementation and refer to our source code for the exact definition.
State-Action DT's (SA-DT) Behavioral cloning SA-DT's are the most common method to generate interpretable policies post-hoc. Hereby, we first train an MLP policy, which is then distilled into DT post-processing step after the training. Specifically, we train the DT on a dataset of expert trajectories generated with the MLP policy. The number of expert trajectories set to 25 which experienced for the distillation and model complexity during preliminary experiments. The 25 expert trajectories result in total of approximately 12500 state-action pairs, varying based on the environment specification.
good trade-off between dataset size
Discretized Soft Decision Trees (D-SDT) SDTs allow gradient computation by assigning probabilities each node. While SDTs exhibit hierarchical structure, they are usually considered as less interpretable, since multiple features are considered a single split and the whole tree is traversed simultaneously (Marton et al. 2024a). Therefore, Silva et al.
28

---

Published as a conference paper at ICLR 2025
(2020) SDTs as policies which discretized post-hoc to allow easy interpretation considering only a single feature at each split. Discretization achieved by employing an argmax to obtain the feature index and normalizing the split threshold based the feature vector. We improved their method by replacing the scaled sigmoid and softmax, with an entmoid and entmax transformationPPtterss. et al.) 2019), resulting in sparse feature selectors with more responsive gradients, as it is common practice Popoy et al. (2019); Chang et al. (2021).
In the following. list the parameter grids used during the hyperparameter optimization (HPO) as well as the optimal parameters selected for each environment. For SYMPOL, SDT and MLP, we optimized the hyperparameters based on the validation reward with optuna Akiba al. 2019 for 60 trials. Thereby, we ensured that the environments evaluated during the HPO were distinct to the environments used for reporting the test performance the rest of the paper. Additionally, we decrease the learning rate if no improvement in validation reward is observed for five consecutive iterations, allowing for finer model adjustments later training stages.
C.I
HPO GridS
Table 13: HPO Grid SYMPOL
hyperparameter
values
learning_rate_actorn wat.taaattt.wwighttt.t.. wwiigts wtwpgggrrtt...ttsspllll values learning rate actor split idx arryy
[0.0001, 0.1]
[0.0001, 0.05]
[0.0001, 0.1]
learning rate actor leaf array
[0.0001, 0.05]
watpiiiggrttttatttrrtt Searningrrat__one
[0.0001, 0.1]
[0.0001, 0.01]
n_update_epoctt reduce Ir n_steps
[0. 10]
(True, False)
{128, 512)
n. envs norm adv
[4. 16]
(True, False)
ent_coef
{0.0, 0.1, 0.2, 0.5)
gac_lambda
{0.8, 0.9, 0.95, 0.99]
gamma
(0.9, 0.95, 0.99, 0.999)
cfef
(0.25, 0.50, 0.75)
max max_ggrad_norm
[None]
SWA
(True)
adamW W
(True)
depth
{7]
minibatch.size
{64)
29

---

Published as a conference paper at ICLR 2025
Table 14: HPO Grid MLP
hyperparameter
values
neurons per layer
[16, 256]
num_layers learning_rate_actoo
[1,3]
[0.0001, 0.01]
learning_rate_cre
[0.0001, 0.01]
minibatch.size
{64, 128, 256, 512]
n.update_epocht n steps envs norm_adv
[1. 10]
{ {128, 512)
[4, 16]
(TTuue, False)
ent_coef
{0.0, 0.1, 0.2, 0.5
gac_lambda
{0.8, 0.9, 0.95, 0.99
gamma
{0.9, 0.95, 0.99, 0.999) {0.25, 0.50, 0.75)
vf.coef
max max_grade max_grad_nmg grad_norm
10.1, 0.5, 1.0, None)
Table 15: HPO Grid SDT
hyperparameter
values
critic
('MLP,, 'SDT')
depth
[4, 8]
temperature
(0.01, 0.05, 0.1, 0.5, 1.0)
learning_rate rattinatgrttactoorrrroo
[0.0001, 0.01]
learning rate_criite
[0.0001, 0.01]
minibatch.soe size
{64, 128, 256, 512)
w.update_epooce
[1, 10]
n_steps
{ [128, 512)
n_envs
[4, 16]
norm_adv
(True, False
ent.coef
{0.0, 0.1, 0.2, 0.5)
gae lambda
{0.8, 0.9, 0.95, 0.99)
gamma
(0.9, 0.95, 0.99, 0.999)
vf_coef
{0.25, 0.50, 0.75)
max max-grad_norm
{0.1, 0.5, 1.0, None)
30

---

Published as a conference paper at ICLR 2025
C.2 BEST HYPERPARAMETERS
Table 16: Best Hyperparameters SYMPOL (Control)
CP
AB
LL
MC-C
PD-C
ent coef
0.200
0.000
0.000
0.500
0.100
gae lambda
0.950
0.950
0.900
0.990
0.800
gamma
0.990
0.990
0.999
0.999
0.999
lcarning couninggrate_actorr
0,048
0.003
0.072
0.000
0.022
lcarning_rate_acten pactor.pplit.vallee values
0.000
0.000
0.001
0.000
0.000
learning rate actor split idx array
0,026
0.052
0.010
0.000
0.010
learning-ran rate actor leaf array
0.020
0.005
0.009
0.028
0.006
learning_raa wwt.ggrteeaaaaa sttteeetttr__ae awttarrooossseeeee log_std learning rate_critie
0.001
0.002
0.021
0.094
0.000
0.001
0.000
0.002
0.002
0.000
max_grad_nom n.envs
None
None
None
None
None
7
n_steps
512
128
512
128
128
n update epochs
7
norm adv
False
False
True
False
True
reduce_Ir
True
True
True
True
False
_f_coef
0.500
0.250
0.500
0.500
0.750
SWA
True
True
True
True
True
adam
True
True
True
True
True
dropout
0,000
0.000
0.000
0.000
0.000
depth
minibatch_size
64
64
64
64
64
n_estimators
Table 17: Best Hyperparameters SYMPOL (MiniGrid)
E-R
DK
LG-5
LG-7
DS
ent_coef
0.100
0.200
0.100
0.100
0.500
gae_lambue
0.990
0.950
0.900
0.900
0.950
gamma
0.900
0.990
0.950
0.990
0.999
learning_rate
0.063
0.042
0.055
0.001
0.036
learning rate actor split values
0.001
0.001
0,006
0.001
0.000
learning rate actor split idx array
0.001
0.001
0.012
0.001
0.009
learning_rate_nee jate__cct__eeaf leforrlaafaao Larray arrra learning_rate wtt.:raeteetrrooo// log_sso learning_rate caw.ggrrtteeeiieeeee
0.003
0.004
0.009
0.008
0.001
0.043
0.021
0.005
0.002
0.038
0.001
0.001
0.001
0.001
0.001
max grad norm nrrm n_envs
None
None
None
None
None
14
14
16
10
n. steps
128
512
512
128
512
n update epochs
8
9
5
4
norm_adv
True
True
True
True
False
reduce.Ir
False
True
True
True
True
vf cocf
0.500
0.500
0.250
0.500
0.250
SWA
True
True
True
True
True
adamW
True
True
True
True
True
dropout
0.000
0.000
0.000
0.000
0.000
depth
7
7
7
7
minibatch_size
64
64
64
64
64
n.estimators
1
1
1
31

---

Published as a conference paper at ICLR 2025
Table 18; Best Hyperparameters MLP (Control)
CP
AB
L.
MC-C
PD-C
adam
False
False
False
False
False
ent_coef
0.200
0.000
0.100
0.100
0.100
gae_ lam_lamdda
0.900
0.900
0.900
0,950
0,950
gamma
0.999
0.990
0.999
0.999
0,990
learning_rat rat._rate_acce acttr
0.001
0.000
0.001
0.005
0.000
learning-rate waapinnggrtteriiiiii ccitic
0.003
0.005
0.003
0.001
0.002
max grad norm
1.000
1.000
0.500
0.100
None
minibatch size
256
256
128
512
128
n_envs
13
13
15
8
n_steps
128
512
512
512
512
p.update_epockl
9.
8
2
2
neurons, per per layer
139
185
46
240
75
norm-ad adv num num layers
False
True
False
True
True
2
2
reduce. Ir
False 0.250
False 0.500
False 0.500
False
False
vf_coef
0.250
0.250
Table 19: Best Hyperparameters MLP (MiniGrid)
E-R
DK
LG-5
LG-7
DS
adamW
False
False
False
False
False
ent_coef
0.100
0.100
0.100
0.100
0.100
gac lambda
0.950
0.900
0.950
0.950
0.990
gamma
0.990
0.900
0.990
0.900
0.990
learning_rai _ate_actor
0.000
0.000
0.002
0.000
0.000
learning rate critic
0.001
0.000
0.003
0.001
0.001
max grad norm
0.100
0.100
0.500
0.100
minibatch Size
64
256
128
512
256
n_envs
13
8
8
12
10
n steps n update epochs
512
256
512
128
128
5
7
neurons_pp Lertoos_per__ayeee ply_layer
112
169
76
28
158
norm_adv
False
True
False
True
True
layersyers
3
reduce. Ir
False
False
False 0.250
False 0.750
False
vf.coff
0.500
0.500
0.500
32

---

Published as a conference paper at ICLR 2025
Table 20: Best Hyperparameters SDT (Control)
CP
AB
LL
MC-C
PD-C
adam
False
False
False
False
False
critic
mlp
mlp
mlp
mip
mlp
depth
7
7
ent_coef
0.000
0.100
0.200
0.000
0,200
gae gae_aa
0.950
0.950
0,990
0,900
0.900
gamma
0.990
0.990
0.999
0.990
0,900
learning rate actor
0.001
0.002
0.001
0.001
0.000
learning rate critic
0.000
0.000
0.001
0.007
0.000
max pax__gad_noom
0.100
0.100
1.000
0.500
0.100
minibatchasize
128
128
128
64
128
n_envs
15
7
14
7
stetes
512
128
512
512
256
n.update.epone
4
10
2
7
norm_ady adv
True False
False
True
False
False
reduce Ir
False
False
False
False
temperature
0.500
0.100
vf_coef
0.500
0.500
0.750
0.250
0.500
Table 21: Best Hyperparameters SDT (MiniGrid)
E-R
DK
LG-5
LG-7
DS
adamW
False
False
False
False
False
critic
sdt
mlp
sdt
sdt
sdt
depth
7
ent_coef
0.100
0.100
0.200
0.100
0.100
gae lambda
0.900
0.950
0.990
0.950
0.900
gamma
0.990
0.900
0.999
0.950
0.950
learning leatinngrrtt__ae rateeacctor
0.004
0.001
0.000
0.002
0.001
learning_rat waatiinggrtteeeee waanng_rat__eittte _eritic
0,000
0.002
0.000
0.005
0.002
max grad norm
0.100
0.100
0.500
0.100
None
minibatch size
512
256
512
256
512
n_envs
10
10
10
13
n_steps
512
256
256
128
512
n_update_epor
5
10
8
4
norm_adv
True
True
True
True
True
reduce_Ir
False
False
False
False
False
temperature
vf coef
0.750
0.750
0.750
0.250
0.750
33


Reviews:
[{"review_id": "dLO1Tq5mM3", "reviewer": "Reviewer_DZ3Q", "summary": "This work employs a tree-based model integrated with on-policy RL, like PPO, to maintain better interpretability with less information loss.\nOn a set of benchmark RL tasks, it demonstrates its superiority over alternative tree-based RL approaches in terms of performance and interoperability.", "questions": "1. line 414- 415: \"In RL, especially for smaller policies, the runtime is mainly determined by the actor-environment interaction...\" - This sentence is hard to follow. Can the authors unpack this sentence?\n2. Did this work simply incorporate GradTree into the PPO framework?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is good, aiming to improve the interpretability of RL, which is a key factor for deployment due to its adoption of DNNs. \n2. The authors try to improve the stability of DTs as a policy via more implemented skills.", "weaknesses": "1. This work evaluates its experiments on a series of toy tasks that could be easily solved. Can it achieve competitive performance in complex tasks, like Mujoco and Atari games?\n2. This work is also hard to follow, the contribution and the method should be highlighted and stated.\n3. Please abatement the contribution of this work. This work is not the first to use DT to represent RL policy, i.e., the work \"MIXRTs: Toward interpretable multi-agent reinforcement learning via mixing recurrent soft decision trees\", arXiv:2305.10091, 2022. They utilized SDT in multi-agent RL complex tasks two years ago.\n4. There are some typos in References.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5JzLEYQSVN", "reviewer": "Reviewer_sgid", "summary": "This paper proposes a differentiable tree-structured model that can be optimized via RL algorithms. The proposed method includes a tree-structure actor and a neural network critic. Experiments are conducted on several RL benchmarks include discrete and continuous action spaces, compared with previous tree-based methods.", "questions": "The reviewer is confused about the information loss experiments, described in Fig 1 and Table 3. Is the test reward on the same environment as the trained one? If so, why is there a performance drop of (Silva et al. (2020))? Did you mean the trained model has a stochastic probability while the test model is greedy that is, selecting the action with the maximum Q?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method can be applied into both discrete and continuous action spaces.\n\n2. The proposed method improves interpretability of RL policies.", "weaknesses": "The reviewer is not convinced with the claim of \"first method that trains a DT via stochastic policy gradient in an end-to-end manner\". Making the learning of DTs differentiable has been investigated for a long time, such as [1]. It is safe to make this claim through a thorough literature review.\n\nThere are some other works focusing on learning a differentiable symbolic policy which is based on first-order logic, such as [2,3], which is not compared and discussed in this paper.\n\nLooks like that defining the syntax and semantics of DTs relies on human knowledge, maybe the reviewer is wrong, which can be clarified during rebuttal.\n\n[1] Treeqn and atreec: differentiable tree-structured models for deep reinforcement learning.\n\n[2] Neural logic reinforcement learning. 2019.\n\n[3] GALOIS: boosting deep reinforcement learning via generalizable logic synthesis. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4n4evGHP0a", "reviewer": "Reviewer_6L9X", "summary": "This work presents SYMPOL, a method for online RL using decision trees as policies. \nThe authors include the GradTree into PPO, thus backpropagating from a neural critic to an arithmetic decision tree-based actor.\nThey evaluate their method on different continuous and discrete environments, and demonstrate that SYMPOL allow to detect and correct misalignments problem that SIMPOL allows to find.", "questions": "* Why is it that the extracted trees are performing so low in comparison to what is reported by the VIPER's authors ?\n* Is there any reason why you are not using the set of Object Centric Atari environments in your evaluation ? It would allow to compare with other deep and interpretable baselines such as NUDGE, INSIGHT, SCoBots, ...\n\nI'm willing to revise my scores if experiments that compare SYMPOL to these interpretable methods on some Atari games are reported. It would allow the reader to compare these methods. Even if SYMPOL slightly underperforms, for interpretability, game scores are not the most important focus.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**The method is well motivated.**\nGBRL, the only interpretable tree-based method that directly learns tree instead of distilling them after training, leads to huge and many trees. \n\n**The potential impact of online learning of trees is huge for interpretable RL.**\nThe online learning of transparent policies could allow for detection of misalignment during training, and potentially correction of the misalignment during training. Distillation methods do not have this advantage.\n\n**The structure of the paper makes it easy to follow**. Most of the paper is clear, in terms of structure, particularly the experimental evaluation. The bold points, together with the clear figures and tables, allow the reader to quickly grasp most of the experimental evaluation, even if it took me a bit of time to understand that SA-DT is an implementation of Dagger/VIPER.", "weaknesses": "**The difference between SYMPOL and other baselines is not clear.**\nThe preliminaries section would benefit for a bigger explanation on GradTree, as its integration to PPO is the main contribution and what distinguish this work from many existing interepretable baselines such as NUDGE, INSIGHT, GBRL and SCoBots ([1]).\nIt also took me quite some time to understand that SA-DT is equivalent to running Dagger or VIPER with their MLP.\n\n**Evaluation seems weird.** It feels like the baselines are underperforming, compared to what is reported in the literature. For example, the D-SDT is underperforming compared to what VIPER reports (with a few node). I don't know why, as it is okay if their baselines would perform worse than distilled ones. Directly learning trees in an online fashion, could allow for e.g. LLM guidance to provide the trees with correct inductive biases, that would avoid e.g. misalignment problems during learning.\n\n**Missing litterature on misgeneralisation (and interpretable RL).**\nOn misgeneralisation, NNs have been shown to systematically misgeneralize on different Atari environment [2], even when the environment is simplified [3], and DT-based policies have been shown to allow to mitigate it [1].\n\n\n[1] Delfosse et al. \"Interpretable concept bottlenecks to align reinforcement learning agents.\" Advances in Neural Information Processing Systems 37 (2024). \n\n[2] Farebrother et al. Generalization and regularization in DQN. arXiv (2018).\n\n[3] Delfosse et al. \"HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning.\" arXiv (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U95EJ4ycj4", "reviewer": "Reviewer_7Ps3", "summary": "Authors propose SYMPOL. SYMPOL alleviates previous challenges of using RL to learn decision tree policies by using a differentiable tree representation. They also increase training stability by collecting longer rollouts at the end of training and using large batch sizes. The paper is well written. However, with the current experiments, it is unclear to me if the good experimental results are due to the use of GradTree or to the stabilizing of the training or both. But in any case, the contribution is not clear.", "questions": "Could you remove fig 2 that does not bring anything to the paper and replace it with appendix results (such as figure 10)? Even better, could you summarize the section 4.1 or GradTree in a schematic and replace the current figure 2 with it please? If you decide to put figure 10 in the main paper instead of the schematic at figure 2, please smooth your curves.\n\nCould you please remove figure 3 and replace it with somehting more useful? I think one does not need a plot to understand your rollout schedule.\n\nCould you write a dedicated parargaph that explains the difference between SYMPOL and Silva et. al. 2020 ? I think your work is actually very similar to their.?\n\nBonus, add CUSTARD (Topin et al 2021) as a baseline in your table 1.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The related work and preliminary sections are excellent. Misgeneralization experiment and information loss experiments are insightful. Topic of interpretability is crucial.", "weaknesses": "There are two main weaknesses to this paper. \n\n1) Contributions. Is the contribution plugging GradTree into PPO ? \n\nI do find the ablation study insightful on why the training process needs strong stabilyzing and digging this further could be a nice contribution. Similarly, it seems to me that Silva et al 2020 is very similar to GradTree in the sense that GradTree is a differentiable formulation of DTs and Silva et al is too. I would have liked a more in depth comparison of the two! \n\n2) Experiments (just the section 5.2). Baselines seem underperforming. \n\nFrom looking at Silva et al 2020 and VIPER 2018, I feel like SA-DT (d=5) and D-SDT should be able to solve CartPole contrary to what you show in your table 1. Also why isn't SA-DT called VIPER? \n\nIf I were you I would rewrite the section 5 centered around two questions: why use direct tree optimization rather than post hoc distillation of a neural net like VIPER? After showing that direct optim might be better than post hoc fitting of trees, you should ask which of the two sota direct optim is better and why: Silva et al or SYMPOL?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y8XNwcEWDB", "reviewer": "Reviewer_6tAh", "summary": "The paper presents SYMPOL, a new approach to learning interpretable policies for reinforcement learning problems by building on recent works facilitating gradient-based training of decision tree architectures. SYMPOL performs much better than prior methods for producing decision tree policies (such as discretizing policies or behavior-cloning a decision tree from a replay buffer), and includes a few tricks for improving the stability of training tree-based models.\n\nSYMPOL is evaluated on several RL problems, largely simpler problems such as mountain car or lunar lander, as well as a MiniGrid environment. Finally, the interpretability of SYMPOL policies is displayed using a case study on policy generalization, in which the authors inspect a learned policy, identify areas for improvement or weaknesses in the policy, and then re-train to fix these weaknesses.", "questions": "* There is no ablation on the automated pruning, but automated pruning is mentioned a few times in the work. What are the effects of this on the resulting policy, and at what point are the node automatically pruned? If they are not pruned during training, do they complicate gradient flow?\n* How is misaligned behavior detected, as in the case study shown in Section 6? It seems that, because the network's parameters are 1D feature vectors, you would need to reverse engineer the input states for each node, then plot out the resulting tree? Is there another way, or is this manually done?\n* Would SYMPOL extend naturally to continuous action spaces or unstructured input (e.g., images)?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper builds on very recent work and adequately cites the prior work, making sure to clearly draw the line between novel contributions and techniques which already exist.\n* The SYMPOL framework is clearly explained, and the tricks used to improve training stability are highlighted (as opposed to being buried in the appendix). I feel like I could re-implement this from the paper alone.\n* The results are compelling. SYMPOL performs quite well (final performance numbers/returns are superior to baselines) while also generalizing well to new versions of the problems (as shown by the Cohen's D experiments)\n* The case study on interpretability is compelling, showing the power of an interpretable RL framework such as SYMPOL. \n\nOverall, the paper is very well written, the framework is clearly described, makes intuitive sense, and does not require significant hand-tuning, and the results and discussion sections are strong.", "weaknesses": "* SYMPOL is mentioned to have 50+ nodes, which could become quite unwieldy or difficult to interpret, but we do not see such examples in the paper. It would be illustrative to see a standard SYMPOL policy in addition to the cleaned up versions in Figures 7/9.\n* While the framework is sound and the results are strong, the overall novelty of the framework is limited. Apart from introducing a dynamically growing replay buffer, the technical contributions are largely a combination of existing techniques.\n* As the paper relies heavily on the contribution of GradTree, it would be helpful to have an overview of that work here, so that the finished paper reads as a more self-contained work.\n* The comparisons across all works are not entirely 1-1, as the number of nodes in each tree is not consistent (however, the authors make note of this in the text).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)