Solving Offline Reinforcement Learning with Decision Tree Regression
Authors: Prajwal Koirala
Cody Fleming
Venue: CoRL 2024 (2024)
Keywords: Offline Reinforcement Learning
Decision Trees
TL;DR: The research introduces Decision Tree-based frameworks for Offline RL, achieving faster training times and comparable performance to traditional methods across various robotic tasks, while emphasizing the interpretability of learned policies.

Abstract: This study presents a novel approach to addressing offline reinforcement learning (RL) problems by reframing them as regression tasks that can be effectively solved using Decision Trees. Mainly, we introduce two distinct frameworks: return-conditioned and return-weighted decision tree policies (RCDTP and RWDTP), both of which achieve notable speed in agent training as well as inference, with training typically lasting less than a few minutes. Despite the simplification inherent in this reformulated approach to offline RL, our agents demonstrate performance that is at least on par with the established methods. We evaluate our methods on D4RL datasets for locomotion and manipulation, as well as other robotic tasks involving wheeled and flying robots. Additionally, we assess performance in delayed/sparse reward scenarios and highlight the explainability of these policies through action distribution and feature importance.

Solving Offline Reinforcement Learning with Decision Tree Regression
Prajwal Koirala lowa State University prajwaleiastate.cm edu
Cody Fleming Iowa State University f flemingc@iastate.e edu
Abstract: This study presents novel approach to addressing offline reinforce- ment learning (RL) problems by reframing them regression tasks that be effectively solved using Decision Trees. Mainly, introduce two distinct frame- works: return-conditioned and return-weighted decision tree policies (RCDTP and RWDTP), both of which achieve notable speed in agent training well as infer- ence, with training typically lasting less than a few minutes. Despite the simpli- fication inherent in this reformulated approach to offline RL, our agents demon- strate performance that is at least on par with the established methods. We evaluate our methods on D4RL datasets for locomotion and manipulation, as well as other robotic tasks involving wheeled and flying robots. Additionally, assess perfor- mance in delayed/sparse reward scenarios and highlight the explainability of these policies through action distribution and feature importance.
Keywords: Offline Reinforcement Learning, Decision Trees
Introduction
There have been many attempts to transcribe a reinforcement learning (RL) problem into a super- vised learning (SL) problem [1, 2, 3, 4. 5]. The motivation behind this transformation primarily stems from key factors. First, by treating RL SI. task, the training process tends show improved stability due reduced susceptibility to the challenges associated with non-stationary targets. This facilitates smoother convergence. Second, the use of a true supervised learning objec- tive enhances data efficiency, preventing it from becoming 'stale" after single learning step. This enables agents to learn more quickly and effectively from their experiences.
However, the distinction between these two learning paradigms (SL and RL) lies in the nature of the feedback signals they utilize. In supervised learning, the error signal, typically represented as loss or cost, is generated from labeled data and minimized through gradient descent algorithms. Conversely, reinforcement learning relies on an evaluation signal provided by the environment, such as reward or return, which require special treatment depending on the setting. For example, Deep Q Networks try to estimate the Q-value of the state-action pair based on the evaluation signal, and Policy Gradient Methods adjust the action probabilities to optimize the policy accordingly [6. 7, 8]. Contemporary state-of-the-arr reinforcement learning approaches often seek encapsulate this specialized treatment through sophisticated loss functions that involve the evaluation signal term, enabling minimization akin to enror signals in supervised learning paradigms.
The traditional online learning paradigm of RL involves iteratively collecting experience by inter- acting with the environment and using that experience to improve the policy [9]. However, recent years, offline reinforcement learning has emerged as a promising approach in robot learning, par- ticularly due to its utilization of previously collected data without the need for continuous online data collection. This circumvents the impracticality and expense associated with continuous data gathering in many settings. By leveraging stationary historical data, offline RL enables agents to leam without direct interaction with the environment during training, thereby transforming large
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

datasets into powerful decision-making engines. While the static nature of the dataset allows of- fline RI to be treated as a supervised learning problem-an apparently advantageous prospect-it also introduces the challenge of covariate shift, the disparity between the data distribution and the learned policy distribution, which necessitates more careful handling [10]. In contrast to offline RL, imitation learning algorithms like Behavior Cloning (BC) and its variants treat the static dataset as labeled data (states as features and actions as labels), often disregarding the evaluation signal [11, 12, 13, 14]. Although Kumar et al. [15], in their comparison of Conservative Q-Learning (CQL (16)) with Behavior Cloning, highlight the capability of offline RI. algorithms to derive effective policies even from suboptimal data, showcasing superiority over BC algorithms in conditions with sparse rewards or noisy datasets, in many cases Behavior Cloning (BC) outperforms offline RL al- gorithms. addition, BCs offer faster training, minimalistic loss function with no requirement of 'special treatment', and are less sensitive to hyperparameter tuning compared to CQL [17].
In this context, we propose transforming reinforcement learning task in an offline setting into simple regression objective, allowing us to leverage decision trees, which have shown strong perfor- mance in regression and classification tasks, often providing a viable alternative to neural networks. By employing an 'extreme' gradient boosting algorithm for regression over actions space, we train the agent's policy as an ensemble of weak policies, achieving rapid training times of mere minutes, not seconds. Besides replacing neural networks as the default function approximators in RL, our paper also introduces two offline RL frameworks: Return Conditioned Decision Tree Policy (RCDTP) and Return Weighted Decision Tree Policy (RWDTP). These methods embody simpli- fied modeling strategies, expedited training methodologies, faster inference mechanisms, and require minimal hyperparameter tuning, while also offering explainable policies. We conduct comprehensive comparisons with prior methods, discuss the use cases of our proposed methods, and also assess our model's performance in scenarios with delayed and sparse rewards using D4RI, benchmark tasks. Finally, we explore the applicabílity of our methods in robot learning and real-time control of dynamical systems, highlighting their potential in practical settings.
Cartpole Angle
0.1
A m
Aglth
Pendulum Angle
RWDTP
RCDTP
0.0
RWDTP RCDTP
-2
100
200
300
400
500
50
100
150
200
Timesteps
Timesteps
(a) Cartpole (Discrete)
(b) Pendulum (Continuous)
Figure 1: Decision Tree Policies in Classical Control Environments with Different Action Spaces. Both methods achieve expert-level returns in both environments using medium-leved demonstration datasets from d3ripy [18], with all trainings completed within second.
2 Preliminaries and Related Works
2.1 Offline Reinforcement Learning
Reinforcement learning (RL) is a mathematical framework for learning- based control, where an agent leams to optimize user-specified reward functions by acquiring near-optimal behavioral skills, represented by policies. This process typically involves iteratively collecting experience by interact- ing with the environment and using that experience to improve the policy.
Offline RL focuses on learning policies from a static dataset without any active data collection. In- stead of obtaining data via environment interactions like in online RI., the agent only has access to a fixed, limited dataset consisting of trajectory rollouts from an arbitrary behavior policy (or policies). The inability of the agent to further explore the environment and collect additional feedback makes it challenging form of learning-bast based decision making and control.

---

The environment in sequential decision decision-making making setting like RL is typically defined as Markov Decision Process (MDP), represented as the tuple (S,A,,, R),, where S is the state space, A is the action space, P is the state transition probability function and R is the reward function. For some 8,8' E S and a E A. P(s'|s, a) x S x A - [0, 1] and R(s, 0, 8') : x A X - R.
A trajectory of time-length T is made up of sequence of states, actions, and rewards (T =< St, 0z. Fr > The reward at any timestep is usually the function of the state, action and the next state. The return at timestep& t, R₁, is the discounted sum of future rewards with discount factor yy: Re(< r2 >,7) = E/ " The return-to-go (RTG) at a timestep is defined as the undiscounted sum of future rewards from that timestep (RTG= Re(< Tr >, = 1)). The goal in RL (including offline RL) to learn a policy that maximizes the expected return in an MDP.
2.2 Decision Tree Regression
Regression analysis aims to establish a relationship between a vector of independent variables (say s) and a vector of dependent variables (say a) by determining appropriate coefficients or parameters to model their association. We seek to minimize the sum of squared residuals hetween a and â, where is the prediction of the model parameterized by Formally, we define the optimization problem as follows: 0" = arg ming Σi= (a; à, (s;; 0))², where, 0" represents the optimal parameters that minimize the sum of the squared differences between the actual values a; and the predicted values á across all observations
,
A powerful approach to solving this regression problem is using regression trees, particularly through the XGBoost algorithm [19, 20, 21J. XGBoost, short for Extreme Gradient Boosting, con- structs an ensemble of decision trees to model the relationship between s and a. This method itera- tively improves the model by minimizing the objective function using a gradient optimization approach. Mathematically, the model output in XGBoost is expressed as the sum of predictions from K individual regression trees: à; Σ (s;), where each fi represents regression tree parameterized by its structure and leaf weights.
The objective function (C) in XGBoost includes a regularization term to penalize model complexity, thereby preventing overfitting. The formal objective function to minimized is:
K
C=
(a: a,)²+ Σ S(f()
1=1
where (2fk)) is the regularization term for the k-th tree, dependent on number of leaves and the weight of the leaves. So each tree fs is optimized to correct the errors of the previous trees, resulting in highly accurate and robust regression model. Furthermore, XGBoost incorporates techniques such as sparsity-aware algorithms for sparse data and weighted quantile sketch for approximate tree learning. It optimizes cache access, data compression, and sharding to build scalable tree-boosting system, making it faster and more scalable than traditional boosting methods, even for large datasets with limited resources 22].
2.3 Related Works
Several approaches have been explored for explainable policy learning, often using decision tree- based methods [23, 24, 25, 26]. Differentiable decision trees (DDTs) is a framework that integrates policy gradient [25], showing that such online-trained policies outperform MLP policies in both performance and confirmed usability and interpretability by a user-study with Likert scale ratings. Similarly, [27] examined neural policies' interpretability through disentanglement, extracting ab- stractions via decision trees based on neuron responses to provide clearer insights into the learned policies of robots.
RWDTP: Advantage-weighted regression (AWR) is an actor-critic algorithm that trains through supervised regression, showing strong performance in both online and offline settings [5]. AWR involves two supervised learning steps: regressing onto target values for the value function and
3

---

regressing onto weighted target actions for the policy. Initially introduced as an off- online algorithm, AWR uses advantage-weighted updates to improve the policy. TD3+BC combines the Twin Delayed Deep Deterministic Policy Gradients (TD3 [28]) algorithm with behavior cloning in the offline setting [29]. adds a simple regression loss to the TD3 loss, regularizing the policy by cloning the behavior observed in the dataset.
Some prevalent approaches focus on reprioritizing the sampling of offline data to improve training. Return-based Data Resample (ReD) adjusts the probability of sampling each transition in the dataset according to its episodic return [30]. BAIL. learns a V-function to select high-performing actions, which are then used to train the policy network using imitation learning [31]. Similarly, Offline Prioritized Experience Replay (OPER) employs priority functions to prioritize highly-rewarding transitions, ensuring they more frequently visited during training [32, 33].
Upside-down reinforcement learning redefines cumulative rewards inputs rather than predictions and adopts 'true' supervised learning objective, ensuring stable and consistent learn- ing targets [3]. This marks a shift in reinforcement learning methodologies by transforming how rewards and learning objectives are utilized [34]. An important work in this line, Decision Trans- former (DT [4]), frames offline RL as a return- setun-conditioned sequence modeling problem [35]. This eliminates the need for bootstrapping for long-term credit assignment and avoids discounting future rewards, preventing short-sighted decisions. Janner et al. [36] utilize diffusion model (DDPM [37]) for trajectory generation, offering flexible behavior synthesis and variable-length planning. The de- noising process allows flexible conditioning: either by using gradients of an objective function to bias plans toward high-reward regions or by conditioning on a specified goal. The guidance of the return-conitionn cotuiticoniiiooinng model is then injected into the reverse sampling stage. Despite being significant milestones in using Generative Models in RI., DT and Diffuser results often do not surpass 'conven- tional' methods like CQL and behavior cloning, even with larger models and increased training and inference times.
RCDTP:
3 RWDTP and RCDTP frameworks
Training off-policy reinforcement learning (RL) algorithm typically involves training critic net- work to approximate the value function while simultaneously training an actor network to maximize this value. However, due to the lack of fresh interactions with the environment, the actor can learn dfftiiibbbttto (OOD) actions that falsely appear to maximize the approximate value function due to estimation errors related to these actions. Contemporary offline RI. algorithms address this overestimation problem through several strategies: regularizing the value function [16, 38, 39], con- straining the action distribution [29, 40, 41], or avoiding the querying of nut-of-distributien actions while training [4, 42, 43]. The decision tree policies (DTPs) trained in this study adopt the third strategy; as they do not rely on value function, no OOD actions are queried while training them.
The policy is deterministic and is conditioned on the state, represented as â = (T;;; The corresponding regression objective is:
RWDTP:
JRW
[an x (Sn;))))
where, N is the number of datapoints in the dataset, R. [0, the discounted sum of future rewards normalized over the dataset and p is hyper-parameter that adjusts the distribution of R, as an exponent. Mathematically, for reward of r, at timestep t in trajectory of length T:
R, minn {
T
R max (R,) minw (R,)
and R
r
RCDTP: In RCDTP, the action is conditioned on the return-to-go go, and timestep in addition to the state. Return to go (RTG) for timestep is the undiscounted sum of future rewards from that

---

timestep (i.e R₁ = E. ra). The policy is represented as â = (T(,, Re, 0).. The corresponding
regression objective is:
N
JRCDTP(T) =
Σ ann (8 Ry, t,;
(3)
Optimal Policy: For both RWDIP and RCDTP, the optimal policy is obtained by solving the unconstrained minimization problem:
Te (-) = argmin,,, (TT))
(4)
neural network aimed at minimizing the cost J as described in equations and 3 typically employs gradient descent or similar method, updating the model parameters through the iterative rule f 0 VV..
However, when using decision trees, the gradients and Hessians need be calculated with respect to the model output (a) at every boosting round [21], rather than the model parameter 0. These gradients and Hessians are then utilized in constructing additional trees until convergence. The output of the decision tree associated with the 4thh boosting round, with observations at step i, is given as at == (8,, R4, i). The final policy will consist of a sum of such outputs,
K
x(-)
Σ n (.).
(5)
for some fixed training budget, K, where each policy x² referred to as weak policy and optimized according to the second-order approximation in equation and using standard techniques for decision-tree splitting and optimization [21].
h argmin
V ( V2--.J- - (()2
2
Policy Training Implementation: In this study, BOOST) algorithm to fit the decision tree policies. RWDTP has four training hyperparameters: the discount factor (7), the power applied to the normalized return, the number of weak policies/trees to be constructed, and the maximum depth of these trees. With the discount factor and return- power both set to 1, RCDTP has only two training hyperparameters. Additionally, RCDTP employs runtime hyperparameter set at the beginning of the episode, the target return, which conditions the policy. This target return is the total return the agent is prompted to collect within the episode and is updated at each step by subtracting the reward obtained in the previous step (R441 R,
the Extreme Gradient Boosting (XG-
Target Target
rt).
4
Experimental Results
4.1 Gym Mujoco Locomotion
This section focuses on the results in Gym-Mujoco environments with the D4RL datasets [44], specifically Walker2d-v2, Hopper-v2, and Halfcheetah-v2. We include comparisons across four kinds of datasets commonly used for benchmarking: medium, medium replay, medium-experr, and expert. Medium and expert datasets are generated from rollouts of 'medium' and 'expert' perfor- mance policies, respectively, trained with Soft Actor- Medium- datasets are created by combining medium datasets with expert datasets, while medium-replay datasets consist of replay buffers generated during the training of medium policies.
Table presents comparison involving recent baselines EDAC and SAC-N, which exhibit superior performance in these tasks, alongside the conventional and widely used baseline, CQL [45, 16]. The mean and standard deviations calculated from evaluations across five seeds. The training is per- formed on hardware comprising Intel Core 10-1390KKF CPU and NVIDIA RTX 4090 GPU

---

Table Results on Gym-MuloCo Locomotion Tasks (All experiments are run by the authors.)
Environment
Dataset
RWDTP
RCDIP
EDAC
SAC-N
CQL
HalfCheetah
Medium
42.11= 1.07
4..44
67.85
69.04
31.72
Medium-Replay
40.03 1.75
40.08 + 0.43
63.52
63.39
44.81
Medium-Expper
85,40 4.64
86.22 3.85
49.73
61.53
27.20
Expert
90.70 1.74
91.24 + 0.45
2.02
1.67
96.65
Hopper
Medhum
55.01 3.51
61.52 -
102.39
9.68
77.42
Medium-Replay
84.10 9.63
83.88 7.25
98.20
101.02
91.70
Medium-Expert
111.24 + 0.84
111.13 1.63
32.55
39.59
27.29
Expert
111.39 + 1.29
110.92 0.76
48.62
1.27
103.29
Walker2d
Medium
79.48
79.62 7.2
90.91
88.01
60.13
Beddium-Repaay
60.96 = 109.16 - 0.37
75.02 8.49
80.55
81.12
82.32
Medium- Expert
85,9918.522
106.69
116.18
44.36
Expert
108.60 0.75
108.15
20.39
1.02
107.58
Average Training Time
20.79 E s.
104.14 1T5.90
2 1hr.
2 Thr.
2 Ihr.
(24 GB), with computational resources for baselines restricted one hour on the GPU ensure fair comparison. Two primary reasons motivate this approach: first, SAC-N, with large num- ber of critics and sufficient gradient steps, can outperform all other methods in locomotion tasks, necessitating equitable comparison; second, our method emphasizes time-efficient training, thus necessitating performance assessment in a resource-limited setting. Interestingly, baseline perfor- mance sometimes improved under these constraints compared to the results reported [46], where complete training results and additional baseline data are also available. RWDTP and RCDTP were trained on CPU, completing training in just few minutes, and demonstrated excellent perfor- mance, particularly in medium-expert and expert datasets. More experiments on enhancing DTP performance in the non-expert regime are discussed in Appendix A.7.
4.2
Gym Robot Manipulation Adroit and Kitchen Tasks
The Adroit domain involves controlling 24-DoF robotic hand, and includes four tasks: pen, door, hammer, and relocate. Each task presents unique challenges that require precise robotic manipula- tion and control. This section includes comparative results from PLAS [47], an offline RL algorithm excelling in manipulation tasks, alongside other baselines [38, 41] employed by PLAS.
Table Results on Adroit Manipulation Tasks (Baseline results are from [17, 46, 47))
Environment
Dataset
RWDTP
RCDTP
EDAC
SAC-N
CQI.
BCQ
PLAS
Pen
Expert
120.65 + 32.81
112.21 25.04
-1.55
87.11
-1.41
114.9
120.7
Door
Expert
105,00 + 0.67
106.42 1 0.25
106.29
-0,33
-0.32
99.0
104.2
Hammer
Expert
126.00 + 2.28
125.23 + 1.02
28.52
28.13
0.26
107.2
127.1
Relocate
Expert
110.62 1.83
109.72 + 2.19
71.94
-0.36
-0.30
41.6
106.9
Franka Kitchen is a multitask environment featuring 9-DoF Franka robot placed in kitchen with some common household items. This is sparse-reward manipulation setup with limited baselines for comparison across its three D4RL. datasets: complete, partial, and mixed.
Table 3: Results on Franka Kitchen (Baseline results are from [47]).
Environment Dataset Kitchen
RWDTP
RCDTP
BEAR
BCQ
PLAS
Complete
50.0 +
60.0 = 12.25
0.0
8.1
34,8
Partial
45.0 24.49
40.0 12.25
18.9
Mixed
50.0 + 0.0
45.0 24.5
47.2
8.1
40.8
4.3 Racecar Gym and Pybullet Drones
Figure 2a compares RCDTP with the Decision Transformer (DT [4]) in the Fltenth racing scenario where the total rewards is measured based on the percentage progress made on the specific race- track/environment [48, 49]. Both return-conditioned methods are trained exclusively on dataset from the Austria environment and their performance tested in other racetracks. They rely solely on lidar and velocity observation without prior track-specifie knowledge. RCDTP outperforms the Decision Transformer in this seen-to-unseen transfcrability test.

---

100
RCDTP
1.25-
Tota Rewards
80
1.00
0.50
Dataset
0.25-
NWO17 (1.0)
RVDTP 10.9)
0.00
50
150
Austria Barcelona Berlin
Columbia
Torino Treitistrasse
Timestep
(a) Racecar Gym
(b) Pybullet Drones
Figure 2: Decision Tree Policies Applied to Wheeled and Flying Robots for Assessment of Zero- Shot Transfer the Learned Policy. (a)
Different Fltenth Racetracks. (b) Goal-conditioning for Different Heights Using RWDTP in the Pybullet Drones Simulation.
Figure 2b demonstrates the Pybullet drones simulation, where the objective is to hover at a certain height [50]. Although the observation space is 27-dimensional, only the height (z dimension) is shown for clarity. With suboptimal demonstration dataset for hovering at Im, we investigate whether the same dataset can train decision tree policy to make the drone hover at different height. By slightly modifying the observation space of RWDTP to incorporate the goal height, the policy can successfully transfer hover at different heights without additional data collection, showcasing the generalization ability of decision policies.
5
Discussions
5.1 How do decision tree policies perform delayed reward scenario?
In delayed reward scenarios where the agent receives 0 reward for all timesteps except the last one when it receives the cumulative reward, excelling requires strong long-term planning and effective of past experiences. Unlike Q-learning variants, Behavior Cloning (BC) is agnostic to this delayed-reward transformation, and our experimental results in show that RWDTP also remains nearly invariant, DTP-BC represents behavior cloning with decision trees and BC results are from the work of Tarasov et al. [17].
Table 4: Performance drop on delayed rewards scenarios
Dataset
Hopper-M M
Hopper-MI
Hopper-MF:
Hopper-
RWDTP
33.96 = 6.01
12%
78.89
20.58
66%
110.37
0.64
= 0.64
11%
110.67 + 0.29
1%
RCDTP
49.78
1.21
110
52.91 1 10.68
137%
46.65
1 6.77
458%
111.92 1 0.82
1 1%%
BC
53.51
0%
29.81
1: 2.07
0%
52.30
110.85 4: 1.02
100% DTP-BC
49.33 4.04
0%
23.08
8.46
0%
53.26
0.784
5.2
Are decision tree policies only effective BC?
In the maze2d environment, a rearse-reward gym robotics environment, both RCDTP and RWDTP
significantly outperform the Behavior Cloning (BC) variations, including Decision Transformer (DT) and behavior cloning with decision trees (DTP-BC), demonstrating superior long-term plan- ning and utilization past experiences (table 5).
Table 5: Results on Maze2d Against %BC
Invaroment
Dataset
RWDIF
RCDIP
100% DCP
100% BC
51% DC
10% DC
DT
Medium
56.51 = 15.8
63.37 +: 56.02
11.61 : 8.77
0.79
11.15 8.06
14.25 2 2.33
31.71
26.33
Lange
76.96 20.57
73,66 1. 26.86
3,85 8.94
2.6
-4.97
11.32
2:110
35.66 4: 28.20
5.3
How fast are decision tree policies in comparision to Sequence Modeling counterparts?
Table 6 presents a comparative analysis our regression-basee methods versus sequence modeling approaches, specifically Decision Transformer (DT [4]) and Trajectory Transformer (TT [43]). This comparison in Hopper expert datasets evaluates performance the same seed, focusing on training time, inference time, and normalized returns. To match the performance of the regression-base

---

models, DT was trained for 30k gradient steps and TT for 10 epochs. Our key findings show that the CPU training time for RWDTP and RCDTP is less than 1% of the GPU training time for DT and TT. In average, the training time for decision tree polices are less than the inference time for TT on the same CPU device. Appendix A.6 includes similar comparisons to other baselines.
Table 6: Training and Inference Time Comparison Against DT and TT
Dataset
Hopper Expert
Method
RWDTP
RCDTP
DT
TT
Training Device
CPU
CPU
GPU
GPU
Training Time
9.64s
11.36s
2896.67s
1961.87s
1.94s
2.09s
34.29s
1.34s
Inference Device
CPU
CPU
CPU
CPU
Inference Time (j)
3.15e-4s
2.57e-4s
2.30e-2s
12.838
(0)
1.29c-3s
5.63e-4s
3,63e-2s
1.11s
Normalized Returns
111.81
113.68
116.25
110.82
5,4 Do RCDTP and RWDTP learn to prioritize similar features and predict similar actions?
Figure 3a shows the distribution the action distance (Ila -a (2)l2)) evaluated over the observations in the Hopper-Expert dataset, where and are two different policies/sourees. Our empirical ex- periments indicate that RCDTP and RWDTP may learn different policies depending on the dataset type, as they are conditioned different inputs despite using similar regression objectives. How- the features that they learn to prioritize are fairly consistent. crattre-importani analysis in Hopper observation space reveals that both models prioritize the angular velocity and the angle of the foot joint the most when predicting the actions (figure 3b). These features closely correspond to the torque applied to the foot rotor in the action space.
20000
Weight and rank assigned by: RWDTP RCDTP
RWDTP-Date Datasst
0.3
Frequency
15000
RCDTP-Dataset
10000
RWDTP-RCDTP
Weights
0.2
2
3
5000
0.1
5
6
5
4
8.0
0.4
1.2
1.6
2.0
0.0
Action Distance
Oract
Groot
Žtorso
Ziorso
@torso
(a) Actions Distribution
(b) Feature Importance
Figure 3: Comparison Between Decision Tree Policies in Hopper Expert Dataset
5.5 Limitations
While the rapid training and inference capabilities make decision tree policies particularly suitable for quick experimentation in robot learning and real-time control of dynamical systems, there some notable limitations. Decision Tree Policies primarily applied flat-structured observation spaces, and due to the choice of function approximator, methods may not extend to more com- plex data modalities such as images and text. Future work could explore integrating these data types to enhance robot perception and decision-making.
Additionally, human behavior often multimodal, with diverse actions in similar contexts, which models may not fully capture generative models do. Handling these multimodal datasets through energy-based methods remains viable future direction. Furthermore, our training process is conducted offline, relying pre-collected datasets. Due to the inherent characteristics of decision trees, which lack adjustable weights like those in neural networks, online training involving the construction of additional trees each round appears impractical.
Despite these limitations, believe that the proposed methods represent significant advancement in integrating piaression-bassee techniques into explainable offline RL. offering a robust foundation for future research and application in robot learning and control.

---

References
[1] D. Ghosh, A. Gupta, A. Reddy, J. Fu, C. Devin, B. Eysenbach, and S. Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088 2019.
[2] A. Kumar, X. B. Peng, and S. Levine. Reward-conditioeed 1912. 13465, 2019.
policies.
arXiv preprint
[3] J. Schmidhuber. Reinforcement learning upside down: Don't predict rewards-just map them
to actions. arXiv preprint arXiv:19/2.02875, 2019.
[4] I.. Chen, K. Lu, A. Rajeswaran, K. Leee A. Grover, M. Laskin, Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances 2021.
in neural information processing systems,
[5] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. devantage-weighted regression: Simple and scalable off off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
[6] A. G. Barto and T. G. Dietterich. Reinforcement learning and its relationship to supervised learning. Handbook of learning and approximate dynamic programming, 10-9780470544785,, 2004.
[7] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for rein- forcement learning with function approximation. Advances in neural information processing systems. 12, 1999.
[8] V. Mnih, K. Kavukcuoglu, D. Silver. A. Rusu, Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, al. Human-level control through deep rein- forcement learning. nature, 518(7540):529-538 -33, 2015.
[9] R. S. Sutton, A. G. Barto, et al. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.
[10] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[11] Kober, A. Bagnell, and J. Peters. Reinforcement learning in robotics: survey. The International Journal of Robotics Research, 32(1 2013.
[12] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2):1-179, 2018.
[13] Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018.
[14] C. G. Atkeson and S. Schaal. Robot learning from demonstration. ICML, volume 97, pages 12-20, 1997.
[15] A. Kumar, J. Hong, A. Singh, and S. Levine. When should prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05500 2022.
[16] A. Kumar, Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning, Advances Neural Information Processing Systems, 33:1179-1191 2020.
[17] D. Tarasov, A. Nikulin, D. Akimov, V. Kurenkov, and S. Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. Advances in Neural Information Processing Sys- tems, 36, 2024.
[18] T. Seno and M. Imai. d3rlpy: An offline deep reinforcement learning library. Journal of Machine Learning Research, 23(315):1-20, 2022.

---

[19] W.-Y. Loh. Classification and regression trees. Wiley interdisciplinary reviews: data mining and krowledge discovery, 1(1):14-23, 2011.
[20] J. H. Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
[21] Chen and C. Guestrin. Xghoost: A scalable tree boosting system. Proceedings of the 22nd aem sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[22] D. McElfresh, S. Khandagale, J. Valverde, V. Prasad C, G. Ramakrishnan, M. Goldblum, and C. White. When do neural nets outperform boosted trees on tabular data? Advances Neural Information Processing Systems, 36, 2024.
[23] D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
[24] A. Suárez and J. F. Lutsko. Globally optimal fuzzy decision trees for classification and regres- sion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(12):127-11111 1999.
[25] A. Silva, T. Killian, I. D. J. Rodriguez, S.-H. Son, and M. Gombolay. Optimization meth- ods for interpretable differentiable decision trees in reinforcement learning. arXiv preprint arXiv:1903.09338 2019.
[26] Ding, P. Hernandez-Leal, G. Ding, C. Li, and Huang. Cdt: Cascading decision trees for explainable reinforcement learning. arXiv preprint arXiv:2011.07553, 2020.
[27] T.-H. Wang, W. Xiao, T. Seyde, R. Hasani, and D. Rus. Measuring interpretability of neural policies of robots with disentangled representation. In Conference on Robot Learning, pages 602-641, PMLR, 2023.
[28] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[29] S. Fujimoto and S. S. Gu. A minimalist approach offline reinforcement learning. Advances in neural information processing systems, 14:20132-20144 2021.
[30] Y. Yue, B. Kang, X. Ma, Z. Xu, G. Huang, and S. Yan. Boosting offline reinforcement learning via data rebalancing. arXiv preprint arXiv:2210.092411, 2022.
[31] X. Chen, Z. Zhou, Z. Wang, C. Wang, Y. Wu, and K. Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33:18353-18363, 2020.
[32] Y. Yue, B. Kang, X. Ma, G. Huang, S. Song, and S. Yan. Offline prioritized experience replay. arXiv preprint 2306.0520. 2023.
[33] Y. Yue, B. Kang, Ma, G. Huang. S. Song, and Y. Shuicheng. Decoupled prioritized resam- pling: Advancing offline rl with improved behavior policy. 2023.
[34] R. K. Srivastava, P. Shyam, F. Mutz, W. Jaskowski, and J. Schmidhuher. Training agents using 2019.
upside-down reinforcement learning. arXiv preprint
[35] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, E.. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[36] M. Janner, Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.00 2009.999,, 2022.
10

---

[37] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[38] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
[39] Kostrikov, R. Fergus, Tompson, and Nachum, Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783. PMLR, 2021.
[40] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911. 1361, 2019.
[41] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[42] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021.
[43] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. Advances neural information processing systems, 14:1273-1286, 2021.
[44] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint TrXii:2004.072//9 2020.
[45] G. An, S. Moon, J.-H. Kim, and H. O. Song. Uncertainty- -bssed offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436- 7447, 2021.
[46] clean offline reinforcement learning, 2022. URL UKLLttpss//gihhh....mmmtttttttte CORL.
[47] W. Zhou, S. Bajracharya, and D. Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, pages 1719-1735. PMLR, 2021.
[48] M. O'Kelly, H. Zheng, D. Karthik, and R. Mangharam. Fitenth: An open-source evalua- tion environment for continuous control and reinforcement learning. Proceedings of Machine Learning Research, 123, 2020.
[49] A. Brunnbauer. Racecar gym, 2021. URL http:///ss.tttac.cooaaaaaeerarer
[50] Panerati, H. Zheng, S. Zhou, Xu, A. Prorok, and A. P. Schoellig. Learning to fly gym environment with pybullet physics for reinforeement learning of multi-agent quadcopter control. In 202/ IEFF/RS/ International Conference Intelligent Robots and Systems (IROS), pages 7512-7519. IEEE, 2021.
[51] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 170.006347 2017.
[52] H. Chen, C. Lu, C. Ying, H. Su, and J. Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling, arXiv preprini urXiv:2209. 14548, 2022.
[53] P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. Idql: Implicit q-learning as actor-critic method with diffusion policies. arXiv preprint vrXiv:2304.00577 2023.
[54] S. Park, K. Frans, S. Levine, and A. Kumar. Is value learning really the main bottleneck in offline rl? arXiv preprint arXiv:2406.0985 2024.

---

A Appendix
A.1 XGBoost algorithm for state-action regression
Let D = =s, i= denote the training dataset, where S; represents the feature vector of instance i and a; denotes its corresponding true target value. In XGBoost, the model output à is determined by aggregating predictions from individual regression trees, denoted as a; = >x=ı (s.(s.). where each f& represents a regression tree parametrized by its structure and Ieaf weights. Any tree f& is optimized to correct the errors of the previous trees, and also referred to as a weak policy this paper. To curb overfitting, XGBoost's objective function (C) integrates regularization term aimed penalizing model complexity.
K ( = Σ (a, az)² + Σ (((
==1
where 0 represents the parameters of the model, a; is the true target value of instance i, and à; is the predicted value. The regularization term St(fa) for the k-th tree is typically defined as:
Tx 2
O(fk) = yTx +
(8)
kj
Here, T& denotes the number of leaves in the k-th tree, Wky represents the weight of the j-th leaf, ? parameter controlling model complexity by penalizing the number of leaves, and À controls L.2 regularization on leaf weights.
The optimization process in XGBoost proceeds by sequentially adding trees fs to fit the residual value that best reduce the objective function. The t-th tree is built to minimize the following objec- tive:
Σ i=1
g.f((ss)) + + + ?(fi)
(9)
where 9: and h2 are the first and second-order gradients of the loss function with respect to the prediction at instance
dL
JCC
9:
oa (a!--))"
h,
(10)
where, alt (t-1) represents the predicted value of instance using the ensemble of t trees,
A.2 Policy Training
Decision tree policy training involves iteratively improving the ensemble model through a series of gradient-boosting steps and detailed in algorithm At each iteration, the algorithm computes the first and second-order gradients (gradients and hessians) of the loss function with respect to the current model's predictions. These gradients and hessians guide the construction of new decision tree (a weak policy) that fits the residual enrors of the current model. This new tree is then added to the ensemble, progressively refining the model's prediction. This iterative process continues until the specified number estimators, K, is reached, thereby optimizing the model by sequentially minimizing the objective function.
Decision Tree Policies are generally not very sensitive to hyperparameters. Additionally, the training process is swift, making the tuning process easy and efficient if needed. For Gym-Mujoco locomo- tion tasks, RWDTP typically uses 100 estimators with tree depth of 11, while RCDTP employs 1000 estimators and a tree depth of 6. We have found these values to be effective starting points for a range of tasks and often adjust the number of estimators based on dataset size. For example, adroit tasks, which involve smaller datasets, only require 50 estimators for RWDTP. Nonetheless, the configuration of 100 estimators and a tree depth of 11 for RWDTP, and 1000 estimators with tree depth of 6 for RCDTP, offer robust baseline settings for hyperparameter tuning across most tasks.
12

---

Algorithm 1: Decision Tree Policy Training
Input States s, Actions a, Target Returns R. Timesteps t (for RCDTP)
Input : States S, Actions a. Powered Return Weights RP (for RWDTP)
Output: Trained Decision Tree Policy
Tree Hypeepaaaameters: Number of estimators K, Maximum Tree Depth D
1. Preprocess Data
For RCDTP: X.train + concatenate(s, R, t), y-train a
For RWDTP: X X_train t s, y_train + a
Initialize Model
For RWDTP:
objective + squared error weighted by Rp
For RCDTP: objective squared error
model < Regressor(objjetivee K, D)
Train Model
Function fit (model, X train, train):
for < 1 to N do
1. Compute gradients 9i and hessians h;
2. Construct new decision tree fit the residuals
3. Add the new tree the model ensemble
end return model
model fit(model, X train, y train)
In addition to the hyperparameters related to estimators and tree depth, RWDTP incorporates an additional hyperparameter, p. which plays a role in weighting the samples based on their normalized return. Specifically, the dataset-normalized return used with an exponential function of p as weight (w Rt),, providing a measure of how valuable each sample is relative others in the dataset. Figure 4 illustrates the impact of this weighting mechanism, controlled by p on normalized returns for different Walker2d datasets. The hyperparameter p allows for adjusting this weighting, where higher values of amplify the prioritization of higher-return samples, effectively skewing the contribution of these samples during training. This enables the model to focus more on higher-return demonstrations. When p is set to 0, the RWDTP method reduces to pure behavior cloning, where all samples contribute equally, regardless their return.
80
Nemaltzed
Walker2d Expert
40
Walber2d Medium
Walborz2 Mediuum Baplay
Whlker Medium Expert
Betum Powerr (p)
Figure 4: Impact of Hyperparameter on Normalized Returns During Evaluation
A.3 RCDTP: Modeling of Returns Distribution
To assess RCDTP's efficacy in modeling returns distributions, we generate plots illustrating the accumulated actual returns achieved by the agent when conditioned on specific target RTG values. For comparison, we also include the Decision Transformer (DT) return distribution plot for the same dataset. Although Decision Transformer uses extended context length, uses larger model, and takes
13

---

50 times more training time, by visual comparison, it is evident that RCDTP's performance is closer to the desired return values, showing a comparable capacity for modeling of returns.
Hopper Nedium
Hopper Medium Replay 92
Medium Expert
80
Dradit
Orudie
DT
BCOTT
RCD
Qrackt
8
Norma
100
Rerurn
Ratumn
Figure Returns Distributions Comparison Between RCDTP and Decision Transformer
A.4
Explainability of Policies
0.,04
Leaves
d< 00.04
é 0.047
True
False, Missing
i 0.303
0.347
Leaf Value
0.094
00.44
0.016
0.013
0.015
0.019
0.050
0.100
Agent Decision
Figure
Weak Policy Example in Cartpole Environment
Decision trees inherently offer greater explainability for policies. Figure 6 illustrates decision tree from ensemble used to form expert-level policy in the CartPole environment. This specific tree shows that the agent's ' decisions are primarily based on the pole's angle: if the angle is negative, the agent is likely to push the cart left, and if positive, it mostly pushes the cart right. This results in an intuitive and interpretable weak policy. As more trees are added to the ensemble, the policy's strength and robustness increase. Feature importance analysis figure further enhances explain- ability, revealing that pole angle and angular velocity are more critical for stabilizing the pole than cart position and velocity. Similarly, in the pendulum environment, the policy equally prioritizes the cosine and sine components of the pendulum angle, giving substantial weight to all dimensions of the observation space in decision making.
0.4-
0.4
0.3
0.0
0.0
cos
sin(e)
(a) Cartpole
(b) Pendulum
Figure 7: Features Prioritized by RWDTP
We have also visualized the decision boundaries learned by the decision tree policies. For CartPole, the decision boundary (whether to push the cart to the right) visualized in figure 8a based on different pole angles and velocities, while keeping the cart's position and velocity fixed at (0,0). For the pendulum environment, we plot, figure 8b, the predicted torque values over various pendulum angles and angular velocities, providing further insight into the model's action prediction surfaces. These visualizations further demonstrate the explainability of our approach across environments, highlighting how decision tree policies make control decisions.
14

---

Tarque
0.8
6
1.5
0.5
0.0
-10
0.2
-0-
0.2
0.0
2
2.0
(a) Cartpolc
(b) Pendulum
Figure 8: Decision Boundaries/Surfaces Learnt by RWDTP
A.5 Racecar Gym and Pybullet Drones Experiments Details
The offline dataset for the Racecar Gym experiments obtained by implementing controller tailored to the Austria racetrack. This controller receives the raceline to be tracked and determines the steering angle and motor command based on the current heading angle and lookahead points in the raceline. After training offline on a hundred episode demonstrations collected using this controller the Austria racetrack, the agents undergo a zero-shot transfer test on various other racetracks. Both RCDTP and DT agents utilize only Lidar and velocity observations, excluding raceline or map information, predict actions, which include steering angle and motor command. In this environment, agents rewarded based racetrack progress and penalized for collisions, leading to episode termination.
We conducted experiments using the hover-aviary environment in Pybullet drones. The target posi- tion set to [o, 0, 1], with the reward based on the L4 norm distance between the drone positions and the target. The primary objective to maintain a hovering position around Starget minimal deviations the x and y dimensions from the starting state. Initially, we trained Proximal Policy Optimization (PPO [51]) for 100,000 timesteps to obtain 'suboptimal" agent and collected 100 episodes to form an offline demonstration dataset. Subsequently,
with
modified the dimension of the dataset using the transformation Starget - as to incorporate the goal height (Ztarget = 1). After training RWDTP on this dataset with slightly modified observation space, we conducted sim- ulations with Starget values of 1, 0.8, and 0.6. The same RWDTP policy hovered the drone at each target height without any policy retraining, modifications to the reward function, or additional dataset collection.
A.6 Training and Inference Time
Table 7 presents the training times (on GPU) and inference times (on CPU) for 1 million gradient steps across various methods, as recorded in our experiments. For baselines, we compare CQL, Behavioral Cloning (BC), and different variants EDAC with 10 and 50 critic networks. The experiments were conducted on system running Ubuntu 22.04.3 LTS, equipped with 13th Gen Intel Core 19- 1390390KFF processor with 24 cores, an NVIDIA RTX 4090 GPU (24 GB), and 64 GB of DDR5 memory.
Table 7: Training and Inference Time (in seconds)
Method
Training Mean Training Std.
Inference Mean
Inference Std
CQL
6053.3
28.3
3.6c-4
1.3c-4
BC
692.26
13.45
1.04e-4
3.32e-5
EDAC-10
3754.5
39.8
4.2e-4
5.8e-4
EDAC-50
5154.0
62.74
15

---

A.7 Using IQL Critic to Guide Action Selection
In our attempt to improve the performance of decision tree policies (DTPs) on datasets without expert-level demonstrations, we conducted additional experiments using an IQL [42] critic to guide action selection. Unlike traditional methods, IQL allows training the Q function without requiring an actor sample unseen actions, providing distinct advantage in offline learning. Specifically, we integrated a perturbation layer with mean of the predicted action and standard deviation of 0.2 into the output of RCDTP and RWDTP methods. From these perturbed actions, we sampled N = 10 actions and selected the one with the highest Q value. This action selection (or policy extraction) method, based on sampling, has been employed before in [52, 53, 54]. The performance of these IQL-assiited DTPs denoted by the suffix +IQL in the table Although the performance of "plain' decision tree policies is on par with the conventional baselines in the non-expert regime, results of our experiments sometimes show performance improvements up to 60% when using the IQL critic. This also highlights the better performance of decision tree policy prior in comparison to the diffusion BC prior used in IDQL [53].
Table 8: Improved Performance in Non-expert Regime (Baseline results are from [53])
Dataset
RWDTP
RCDIP
RWDTP+IQL.
RCDTP+IQL.
IQL.
IDQL.
CQL
D)T
BC
Hallcheetah Med
42.11
41.44
49.38
48.83
47.4
51.0
44.0)
42.6
43.1
Hopper Med
55.01
61.52
87.04
88.58
66.3
65.4
67.6
63.9
Walker2d Med
79.48
79.62
87.96
83.07
78.3
82.5
72.5
74.0
77.3
Halfcheetah Med-Rep
40.03
40.08
45,39
43.91
44.2
45.9
45.5
36.6
4.3
Hopper Med-Rep Rep
84.10
83.88
89.59
96.27
94.7
92.1
95.0
82.7
27.6
Walker2d Med-Rep
60.96
75.02
78.56
79.53
73.9
85.1
66.6
36.9
Average
60.28
63.59
72.99
73.37
67.47
70.33
65.45
61.68
42.18
To understand the performance gap between decision tree policies and their +IQI. counterparts, we examine the distribution of actions learned by RWDTP and RWDTP+IQL in Hopper-Mediumt where the gap is lower. Figure depicts the
where the gap is high, and distribution of action distance (llapottey Odataset ||2) evaluated over the observations in the dataset. Specifically, we examine the Hopper Medium and Medium Replay scenarios to illustrate the impact of potential overfitting on the performace. RWDTP's distribution of action distances from the dataset is highly skewed towards zero for Hopper-Mediun compared Hopper-Medium Medium Medium-Replaay indicating a lack of As result, when the IQL critic is used to guide the action selection, it shifts this action-distance distribution away from zero and also greatly improves performance.
generalization and potential overfitting within the dataset's 's distribution.
1600
6000
RWDTP
RWDTP
RWDTP+1QL
RWDTP+10L
Frequency
4500
Frequency
1200
800
3000
1500
0.0
0.5
1.0
1.5
2.0
0.0
0.6
1.2
1.8
2.4
3.0
Action Distance From Demonstrations
Action Distance From Demonstrations
(a) Hopper Medium
(h) Hopper Medium Replay
Figure 9: Distribution of Action Distance From Demonstrations
A.8 Additional Feature Importance Analysis
Feature importance analysis can be crucial tool in robot learning, particularly for understanding which input features are most influential in the decision-making process of learned policies. By identifying key features, which is essential for ensuring interoperability across different tasks and environments. In this
better assess the interpretability and robustness of these policies,
16

---

additional study, we applied feature importance analysis to the Expert and Medium datasets in the Hopper and Walker2d environments to evaluate the behavior of the decision tree policies.
Weight and rank assigned by:
Weight and rank assigned by:
0.3
RWDTP
1
RWDTP
RCDTP
RCDTP
Weights
Weights
0.2
2
0.2
2
3
0.1
0.1
5
0.0
0.0
Groot
Groot
ZLorso
Ztorso
Ocorso
Oroot
0
Bieg
Ziors
Ziorso
(a) Hopper Expert
(b) Hopper Medium
Weight and rank assigned by:
Weight and rank assigned by:
0.3
RWDTP
0.3
RWDTP
RCDTP
RCDTP
Weights
Weights
2
0.2
2
3
0.1
5
4
5
0.1
6
11
0.0
leetfoot
Brightfoot
Ztorse
rrightroot.
Berftoott
Bightfoot
fieffoott
@iettthigh
Geettigg
Bietttoot
(c) Walker2d Expert
(d) Walker2d Medium
Figure 10: Feature Importance
Figures 10a through 10d display the ranked importance of key features identified by both meth- ods. Our analysis reveals that despite being trained on different performance datasets, RWDTP and RCDTP tend prioritize similar features when predicting actions. For each environment, the two most important features identified by both methods (DTPs) in both datasets remain the same. The rankings of important features as identified by the two policies in the Hopper datasets are summa- rized in table 9. This kind of consistency can be important for ensuring the interpretability of policies before deploying them to real-world robotic tasks.
Table 9: Feature Importance Rank Assigned in Hopper Datasets
Observation
RWDTP (Espe((Epper) RWDTP (Medium)
RCDTP (Expert)
RCDTP (Medium)
Angular velocity of the foot hinge
Angle of the foot joint
2
2
2
2
Velocity the z-cordiinat of the torso
7-coordinate of
17


Reviews:
[{"review_id": "DuzkO3oUuV", "reviewer": "Reviewer_NHhZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method using Decision Tree Regression to learn optimal and explainable policies from offline data, showing strong performance on D4RL datasets, but raises concerns about potential overfitting and underfitting issues with non-expert data.", "summary_of_recommendation": "I would recommend weak reject and wait for the rebuttal from the author.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 2, "questions_for_rebuttal": "- In the RWDTP method, what is the intuition of using reward as the weighted for the lost?\n - The decision tree policies trained in this paper are based on avoiding out-of-distribution action distribution. \n - Do you have problems with overfitting and underfitting when learning the optimal policy? What if the offline data collected is not from the expert trajectory?  Is it true that our performance is not competitive when we do not have an expert dataset?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ktR5jL2Ip6", "reviewer": "Reviewer_i7UK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes methods to solve offline reinforcement learning problems via decision tree representations, incorporating return information either via conditioning (RCDTP) or by weighting the optimization objective (RWDTP) and running XG-BOOST. The methods are tested on a variety of continuous control offline RL task and display favorable performance, particularly in the expert demonstration regime, while significantly cutting down on training time.", "summary_of_recommendation": "I lean towards accept as the method’s simplicity is a nice departure from the more conventional offline RL setting, with very fast training times that results in favorable policy performance. Quantitative and qualitative analysis could be strengthened for full transparency and higher impact.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "- I might have missed this, but over how many seeds/re-runs were the standard deviation results computed?\n- In the drone scenario, how was the goal height incorporated into the observations (agent height and difference to goal height)? Which initial conditions (agent height) does the expert data contain?\n- At which dimensionality of both the state and action space would the method reach its limits?\n- What are the values for the number of estimators and tree depth for each task?\n- In Algorithm 1, does N refer to K?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "65dCaxEf1F", "reviewer": "Reviewer_c4Yz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Using gradient-boosted decision trees to solve offline RL tasks.", "summary_of_recommendation": "Surprisingly strong performance by decision trees, which could generate good discussion at the conference.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "See questions above, particulary in the Major Comments.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers are excited about this work. It dares to go against the current flow of using neural networks for low-dimensional tasks, and demonstrates strong merits of using decision trees. It achieves superior performance and yields explainable policies. This excitement is somewhat mitigated by a long list of technical questions by all reviewers, including recurring questions around reliance on expert demonstrations.

The authors provide additional detail and clarifications in their rebuttal. Acceptance would be on faith that these will be integrated into the paper as promised.

Some particular strengths of this paper:
- well written and easy to follow,
- strong performance on diverse robotic tasks,
- the method provides explainable policies,
- fast training times.

Questions mostly concerned technical details and were largely answered convincingly by the authors.
Decision: Accept