Solving Offline Reinforcement Learning with Decision Tree Regression

Authors: prajwal  koirala
cody  fleming
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 21:25:10

Abstract: This study presents a novel approach to addressing offline reinforcement learning (RL) problems by reframing them as regression tasks that can be effectively solved using Decision Trees. Mainly, we introduce two distinct frameworks: return-conditioned and return-weighted decision tree policies (RCDTP and RWDTP), both of which achieve notable speed in agent training as well as inference, with training typically lasting less than a few minutes. Despite the simplification inherent in this reformulated approach to offline RL, our agents demonstrate performance that is at least on par with the established methods. We evaluate our methods on D4RL datasets for locomotion and manipulation, as well as other robotic tasks involving wheeled and flying robots. Additionally, we assess performance in delayed/sparse reward scenarios and highlight the explainability of these policies through action distribution and feature importance.

Main paper: 
 introduction: 
 There have been many attempts to transcribe a reinforcement learning (RL) problem into a supervised learning (SL) problem [1, 2, 3, 4, 5] . The motivation behind this transformation primarily stems from two key factors. First, by treating RL as a SL task, the training process tends to show improved stability due to reduced susceptibility to the challenges associated with non-stationary targets. This facilitates smoother convergence. Second, the use of a true supervised learning objective enhances data efficiency, preventing it from becoming 'stale' after a single learning step. This enables agents to learn more quickly and effectively from their experiences. However, the distinction between these two learning paradigms (SL and RL) lies in the nature of the feedback signals they utilize. In supervised learning, the error signal, typically represented as loss or cost, is generated from labeled data and minimized through gradient descent algorithms. Conversely, reinforcement learning relies on an evaluation signal provided by the environment, such as reward or return, which require special treatment depending on the setting. For example, Deep Q Networks try to estimate the Q-value of the state-action pair based on the evaluation signal, and Policy Gradient Methods adjust the action probabilities to optimize the policy accordingly [6, 7, 8] . Contemporary state-of-the-art reinforcement learning approaches often seek to encapsulate this specialized treatment through sophisticated loss functions that involve the evaluation signal term, enabling minimization akin to error signals in supervised learning paradigms.The traditional online learning paradigm of RL involves iteratively collecting experience by interacting with the environment and using that experience to improve the policy [9] . However, in recent years, offline reinforcement learning has emerged as a promising approach in robot learning, particularly due to its utilization of previously collected data without the need for continuous online data collection. This circumvents the impracticality and expense associated with continuous data gathering in many settings. By leveraging stationary historical data, offline RL enables agents to learn without direct interaction with the environment during training, thereby transforming large datasets into powerful decision-making engines. While the static nature of the dataset allows offline RL to be treated as a supervised learning problem-an apparently advantageous prospect-it also introduces the challenge of covariate shift, the disparity between the data distribution and the learned policy distribution, which necessitates a more careful handling [10] . In contrast to offline RL, imitation learning algorithms like Behavior Cloning (BC) and its variants treat the static dataset as labeled data (states as features and actions as labels), often disregarding the evaluation signal [11, 12, 13, 14] . Although Kumar et al. [15] , in their comparison of Conservative Q-Learning (CQL [16] ) with Behavior Cloning, highlight the capability of offline RL algorithms to derive effective policies even from suboptimal data, showcasing superiority over BC algorithms in conditions with sparse rewards or noisy datasets, in many cases Behavior Cloning (BC) outperforms offline RL algorithms. In addition, BCs offer faster training, minimalistic loss function with no requirement of 'special treatment', and are less sensitive to hyperparameter tuning compared to CQL [17] .In this context, we propose transforming a reinforcement learning task in an offline setting into a simple regression objective, allowing us to leverage decision trees, which have shown strong performance in regression and classification tasks, often providing a viable alternative to neural networks. By employing an 'extreme' gradient boosting algorithm for regression over actions space, we train the agent's policy as an ensemble of weak policies, achieving rapid training times of mere minutes, if not seconds. Besides replacing neural networks as the default function approximators in RL, our paper also introduces two offline RL frameworks: Return Conditioned Decision Tree Policy (RCDTP) and Return Weighted Decision Tree Policy (RWDTP). These methods embody simplified modeling strategies, expedited training methodologies, faster inference mechanisms, and require minimal hyperparameter tuning, while also offering explainable policies. We conduct comprehensive comparisons with prior methods, discuss the use cases of our proposed methods, and also assess our model's performance in scenarios with delayed and sparse rewards using D4RL benchmark tasks. Finally, we explore the applicability of our methods in robot learning and real-time control of dynamical systems, highlighting their potential in practical settings. Both methods achieve expert-level returns in both environments using medium-level demonstration datasets from d3rlpy [18] , with all trainings completed within a second. 
offline reinforcement learning: 
 Reinforcement learning (RL) is a mathematical framework for learning-based control, where an agent learns to optimize user-specified reward functions by acquiring near-optimal behavioral skills, represented by policies. This process typically involves iteratively collecting experience by interacting with the environment and using that experience to improve the policy.Offline RL focuses on learning policies from a static dataset without any active data collection. Instead of obtaining data via environment interactions like in online RL, the agent only has access to a fixed, limited dataset consisting of trajectory rollouts from an arbitrary behavior policy (or policies). The inability of the agent to further explore the environment and collect additional feedback makes it a challenging form of learning-based decision making and control.The environment in a sequential decision-making setting like RL is typically defined as a Markov Decision Process (MDP), represented as the tuple (S, A, P, R), where S is the state space, A is the action space, P is the state transition probability function and R is the reward function. For some s, s ′ ∈ S and a ∈ A, P(s ′ |s, a) : S × S × A → [0, 1] and R(s, a, s ′ ) : S × A × S → R.A trajectory of time-length T is made up of a sequence of states, actions, and rewards (τ =< s t , a t , r t > T t=0 ). The reward at any timestep is usually the function of the state, action and the next state. The return at timestep t, R t , is the discounted sum of future rewards with discount factor γ: R t (< r t >, γ) = T k=t γ k-t r k . The return-to-go (RTG) at a timestep is defined as the undiscounted sum of future rewards from that timestep (RTG= R t (< r t >, γ = 1)). The goal in RL (including offline RL) is to learn a policy that maximizes the expected return in an MDP.Regression analysis aims to establish a relationship between a vector of independent variables (say s) and a vector of dependent variables (say a) by determining appropriate coefficients or parameters to model their association. We seek to minimize the sum of squared residuals between a and â, where â is the prediction of the model parameterized by θ. Formally, we define the optimization problem as follows: θ * = arg min θ n i=1 (a i -âi (s i ; θ)) 2 , where, θ * represents the optimal parameters that minimize the sum of the squared differences between the actual values a i and the predicted values âi across all observations i = 1, . . . , n.A powerful approach to solving this regression problem is using regression trees, particularly through the XGBoost algorithm [19, 20, 21] . XGBoost, short for Extreme Gradient Boosting, constructs an ensemble of decision trees to model the relationship between s and a. This method iteratively improves the model by minimizing the objective function using a gradient-based optimization approach. Mathematically, the model output â in XGBoost is expressed as the sum of predictions from K individual regression trees: âi = K k=1 f k (s i ), where each f k represents a regression tree parameterized by its structure and leaf weights.The objective function (L) in XGBoost includes a regularization term to penalize model complexity, thereby preventing overfitting. The formal objective function to be minimized is:EQUATIONwhere Ω(f k ) is the regularization term for the k-th tree, dependent on number of leaves and the weight of the leaves. So each tree f k is optimized to correct the errors of the previous trees, resulting in a highly accurate and robust regression model. Furthermore, XGBoost incorporates techniques such as sparsity-aware algorithms for sparse data and weighted quantile sketch for approximate tree learning. It optimizes cache access, data compression, and sharding to build a scalable tree-boosting system, making it faster and more scalable than traditional boosting methods, even for large datasets with limited resources [21, 22] .Several approaches have been explored for explainable policy learning, often using decision treebased methods [23, 24, 25, 26] . Differentiable decision trees (DDTs) is a framework that integrates policy gradient [25] , showing that such online-trained policies outperform MLP policies in both performance and confirmed usability and interpretability by a user-study with Likert scale ratings.Similarly, [27] examined neural policies' interpretability through disentanglement, extracting abstractions via decision trees based on neuron responses to provide clearer insights into the learned policies of robots.RWDTP: Advantage-weighted regression (AWR) is an actor-critic algorithm that trains through supervised regression, showing strong performance in both online and offline settings [5] . AWR involves two supervised learning steps: regressing onto target values for the value function and regressing onto weighted target actions for the policy. Initially introduced as an off-policy online algorithm, AWR uses advantage-weighted updates to improve the policy. TD3+BC combines the Twin Delayed Deep Deterministic Policy Gradients (TD3 [28] ) algorithm with behavior cloning in the offline setting [29] . It adds a simple regression loss to the TD3 loss, regularizing the policy by cloning the behavior observed in the dataset.Some prevalent approaches focus on reprioritizing the sampling of offline data to improve training. Return-based Data Resample (ReD) adjusts the probability of sampling each transition in the dataset according to its episodic return [30] . BAIL learns a V-function to select high-performing actions, which are then used to train the policy network using imitation learning [31] . Similarly, Offline Prioritized Experience Replay (OPER) employs priority functions to prioritize highly-rewarding transitions, ensuring they are more frequently visited during training [32, 33] .RCDTP: Upside-down reinforcement learning redefines cumulative rewards as inputs rather than predictions and adopts a 'true' supervised learning objective, ensuring stable and consistent learning targets [3] . This marks a shift in reinforcement learning methodologies by transforming how rewards and learning objectives are utilized [34] . An important work in this line, Decision Transformer (DT [4] ), frames offline RL as a return-conditioned sequence modeling problem [35] . This eliminates the need for bootstrapping for long-term credit assignment and avoids discounting future rewards, preventing short-sighted decisions. Janner et al. [36] utilize a diffusion model (DDPM [37] ) for trajectory generation, offering flexible behavior synthesis and variable-length planning. The denoising process allows flexible conditioning: either by using gradients of an objective function to bias plans toward high-reward regions or by conditioning on a specified goal. The guidance of the return-conditioning model is then injected into the reverse sampling stage. Despite being significant milestones in using Generative Models in RL, DT and Diffuser results often do not surpass 'conventional' methods like CQL and behavior cloning, even with larger models and increased training and inference times. 
rwdtp and rcdtp frameworks: 
 Training an off-policy reinforcement learning (RL) algorithm typically involves training a critic network to approximate the value function while simultaneously training an actor network to maximize this value. However, due to the lack of fresh interactions with the environment, the actor can learn out-of-distribution (OOD) actions that falsely appear to maximize the approximate value function due to estimation errors related to these actions. Contemporary offline RL algorithms address this overestimation problem through several strategies: regularizing the value function [16, 38, 39] , constraining the action distribution [29, 40, 41] , or avoiding the querying of out-of-distribution actions while training [4, 42, 43] . The decision tree policies (DTPs) trained in this study adopt the third strategy; as they do not rely on a value function, no OOD actions are queried while training them.The policy is deterministic and is conditioned on the state, represented as ât = π(s t ; θ). The corresponding regression objective is:J RW DT P (π) = N n=1 (a n -π(s n ; θ)) 2 Rp n (2)where, N is the number of datapoints in the dataset, Rn ∈ [0, 1] is the discounted sum of future rewards normalized over the dataset and p is a hyper-parameter that adjusts the distribution of Rn as an exponent. Mathematically, for a reward of r t at timestep t in a trajectory of length T :Rt = R t -min N {R n } max N {R n } -min N {R n } and R t = T k=t γ k-t r k RCDTP:In RCDTP, the action is conditioned on the return-to-go, and timestep in addition to the state. Return to go (RTG) for a timestep is the undiscounted sum of future rewards from that timestep (i.e R t = T t r t ). The policy is represented as ât = π(s t , R t , t; θ). The corresponding regression objective is:EQUATIONOptimal Policy: For both RWDTP and RCDTP, the optimal policy is obtained by solving the unconstrained minimization problem:EQUATIONA neural network aimed at minimizing the cost J as described in equations 2 and 3 typically employs gradient descent or a similar method, updating the model parameters θ through the iterative rule θ ← θ -α∇ θ J.However, when using decision trees, the gradients and Hessians need to be calculated with respect to the model output (â) at every boosting round [21] , rather than the model parameter θ. These gradients and Hessians are then utilized in constructing additional trees until convergence. The output of the decision tree associated with the k th boosting round, with observations at step i, is given asâk i := π k (s i , R i , i).The final policy will consist of a sum of such outputs,EQUATIONfor some fixed training budget, K, where each policy π k is referred to as a weak policy and is optimized according to the second-order approximation in equation 6 and using standard techniques for decision-tree splitting and optimization [21] .EQUATION)Policy Training Implementation: In this study, we use the Extreme Gradient Boosting (XG-BOOST) algorithm to fit the decision tree policies. RWDTP has four training hyperparameters: the discount factor (γ), the power applied to the normalized return, the number of weak policies/trees to be constructed, and the maximum depth of these trees. With the discount factor and return-power both set to 1, RCDTP has only two training hyperparameters. Additionally, RCDTP employs a runtime hyperparameter set at the beginning of the episode, the target return, which conditions the policy. This target return is the total return the agent is prompted to collect within the episode and is updated at each step by subtracting the reward obtained in the previous step (R Target t+1 = R Target t -r t ).4 Experimental Results 
gym mujoco locomotion: 
 This section focuses on the results in Gym-Mujoco environments with the D4RL datasets [44] , specifically Walker2d-v2, Hopper-v2, and Halfcheetah-v2. We include comparisons across four kinds of datasets commonly used for benchmarking: medium, medium replay, medium-expert, and expert. Medium and expert datasets are generated from rollouts of 'medium' and 'expert' performance policies, respectively, trained with Soft Actor-Critic. Medium-expert datasets are created by combining medium datasets with expert datasets, while medium-replay datasets consist of replay buffers generated during the training of medium policies.Table 1 presents a comparison involving recent baselines EDAC and SAC-N, which exhibit superior performance in these tasks, alongside the conventional and widely used baseline, CQL [45, 16] . The mean and standard deviations are calculated from evaluations across five seeds. The training is performed on hardware comprising an Intel Core i9-13900KF CPU and an NVIDIA RTX 4090 GPU (24 GB), with computational resources for baselines restricted to one hour on the GPU to ensure a fair comparison. Two primary reasons motivate this approach: first, SAC-N, with a large number of critics and sufficient gradient steps, can outperform all other methods in locomotion tasks, necessitating an equitable comparison; second, our method emphasizes time-efficient training, thus necessitating performance assessment in a resource-limited setting. Interestingly, baseline performance sometimes improved under these constraints compared to the results reported in [46] , where complete training results and additional baseline data are also available. RWDTP and RCDTP were trained on a CPU, completing training in just a few minutes, and demonstrated excellent performance, particularly in medium-expert and expert datasets. More experiments on enhancing DTP performance in the non-expert regime are discussed in Appendix A.7.The Adroit domain involves controlling a 24-DoF robotic hand, and includes four tasks: pen, door, hammer, and relocate. Each task presents unique challenges that require precise robotic manipulation and control. This section includes comparative results from PLAS [47] , an offline RL algorithm excelling in manipulation tasks, alongside other baselines [38, 41] employed by PLAS. Franka Kitchen is a multitask environment featuring a 9-DoF Franka robot placed in a kitchen with some common household items. This is a sparse-reward manipulation setup with limited baselines for comparison across its three D4RL datasets: complete, partial, and mixed. Figure 2a compares RCDTP with the Decision Transformer (DT [4] ) in the F1tenth racing scenario where the total rewards is measured based on the percentage progress made on the specific racetrack/environment [48, 49] . Both return-conditioned methods are trained exclusively on a dataset from the Austria environment and their performance is tested in other racetracks. They rely solely on lidar and velocity observation without prior track-specific knowledge. RCDTP outperforms the Decision Transformer in this seen-to-unseen transferability test. Figure 2b demonstrates the Pybullet drones simulation, where the objective is to hover at a certain height [50] . Although the observation space is 27-dimensional, only the height (z dimension) is shown for clarity. With a suboptimal demonstration dataset for hovering at 1m, we investigate whether the same dataset can train a decision tree policy to make the drone hover at a different height. By slightly modifying the observation space of RWDTP to incorporate the goal height, the policy can successfully transfer to hover at different heights without additional data collection, showcasing the generalization ability of decision tree policies. 
how do decision tree policies perform in delayed reward scenario?: 
 In delayed reward scenarios where the agent receives 0 reward for all timesteps except the last one when it receives the cumulative reward, excelling requires strong long-term planning and effective use of past experiences. Unlike Q-learning variants, Behavior Cloning (BC) is agnostic to this delayed-reward transformation, and our experimental results in 4 show that RWDTP also remains nearly invariant. DTP-BC represents behavior cloning with decision trees and BC results are from the work of Tarasov et al. [17] . In the maze2d environment, a sparse-reward gym robotics environment, both RCDTP and RWDTP significantly outperform the Behavior Cloning (BC) variations, including Decision Transformer (DT) and behavior cloning with decision trees (DTP-BC), demonstrating superior long-term planning and utilization of past experiences (table 5 ). Table 6 presents a comparative analysis of our regression-based methods versus sequence modeling approaches, specifically Decision Transformer (DT [4] ) and Trajectory Transformer (TT [43] ). This comparison in Hopper expert datasets evaluates performance on the same seed, focusing on training time, inference time, and normalized returns. To match the performance of the regression-based models, DT was trained for 30k gradient steps and TT for 10 epochs. Our key findings show that the CPU training time for RWDTP and RCDTP is less than 1% of the GPU training time for DT and TT. In average, the training time for decision tree polices are less than the inference time for TT on the same CPU device. Appendix A.6 includes similar comparisons to other baselines. Figure 3a shows the distribution of the action distance (||a A -a B || 2 ) evaluated over the observations in the Hopper-Expert dataset, where A and B are two different policies/sources. Our empirical experiments indicate that RCDTP and RWDTP may learn different policies depending on the dataset type, as they are conditioned on different inputs despite using similar regression objectives. However, the features that they learn to prioritize are fairly consistent. Feature-importance analysis in Hopper observation space reveals that both models prioritize the angular velocity and the angle of the foot joint the most when predicting the actions (figure 3b ). These features closely correspond to the torque applied to the foot rotor in the action space. While the rapid training and inference capabilities make decision tree policies particularly suitable for quick experimentation in robot learning and real-time control of dynamical systems, there are some notable limitations. Decision Tree Policies are primarily applied to flat-structured observation spaces, and due to the choice of function approximator, our methods may not extend to more complex data modalities such as images and text. Future work could explore integrating these data types to enhance robot perception and decision-making.Additionally, human behavior is often multimodal, with diverse actions in similar contexts, which our models may not fully capture as generative models do. Handling these multimodal datasets through energy-based methods remains a viable future direction. Furthermore, our training process is conducted offline, relying on pre-collected datasets. Due to the inherent characteristics of decision trees, which lack adjustable weights like those in neural networks, online training involving the construction of additional trees in each round appears impractical.Despite these limitations, we believe that the proposed methods represent a significant advancement in integrating regression-based techniques into explainable offline RL, offering a robust foundation for future research and application in robot learning and control.A.1 XGBoost algorithm for state-action regression Let D = < s i , a i > N i=1 denote the training dataset, where s i represents the feature vector of instance i and a i denotes its corresponding true target value. In XGBoost, the model output â is determined by aggregating predictions from K individual regression trees, denoted as âi = K k=1 f k (s i ), where each f k represents a regression tree parametrized by its structure and leaf weights. Any tree f k is optimized to correct the errors of the previous trees, and is also referred to as a weak policy in this paper. To curb overfitting, XGBoost's objective function (L) integrates a regularization term aimed at penalizing model complexity.EQUATIONwhere θ represents the parameters of the model, a i is the true target value of instance i, and âi is the predicted value. The regularization term Ω(f k ) for the k-th tree is typically defined as:Ω(f k ) = γT k + 1 2 λ T k j=1 w 2 kj (8)Here, T k denotes the number of leaves in the k-th tree, w kj represents the weight of the j-th leaf, γ is a parameter controlling model complexity by penalizing the number of leaves, and λ controls L2 regularization on leaf weights.The optimization process in XGBoost proceeds by sequentially adding trees f k to fit the residual value that best reduce the objective function. The t-th tree is built to minimize the following objective:EQUATIONwhere g i and h i are the first and second-order gradients of the loss function with respect to the prediction at instance i.EQUATIONi represents the predicted value of instance i using the ensemble of t -1 trees.Decision tree policy training involves iteratively improving the ensemble model through a series of gradient-boosting steps and is detailed in algorithm 1. At each iteration, the algorithm computes the first and second-order gradients (gradients and hessians) of the loss function with respect to the current model's predictions. These gradients and hessians guide the construction of a new decision tree (a weak policy) that fits the residual errors of the current model. This new tree is then added to the ensemble, progressively refining the model's prediction. This iterative process continues until the specified number of estimators, K, is reached, thereby optimizing the model by sequentially minimizing the objective function.Decision Tree Policies are generally not very sensitive to hyperparameters. Additionally, the training process is swift, making the tuning process easy and efficient if needed. For Gym-Mujoco locomotion tasks, RWDTP typically uses 100 estimators with a tree depth of 11, while RCDTP employs 1000 estimators and a tree depth of 6. We have found these values to be effective starting points for a range of tasks and often adjust the number of estimators based on dataset size. For example, adroit tasks, which involve smaller datasets, only require 50 estimators for RWDTP. Nonetheless, the configuration of 100 estimators and a tree depth of 11 for RWDTP, and 1000 estimators with a tree depth of 6 for RCDTP, offer robust baseline settings for hyperparameter tuning across most tasks. In addition to the hyperparameters related to estimators and tree depth, RWDTP incorporates an additional hyperparameter, p, which plays a role in weighting the samples based on their normalized return. Specifically, the dataset-normalized return is used with an exponential function of p as a weight (w = Rp ), providing a measure of how valuable each sample is relative to others in the dataset. Figure 4 illustrates the impact of this weighting mechanism, controlled by p on normalized returns for different Walker2d datasets. The hyperparameter p allows for adjusting this weighting, where higher values of p amplify the prioritization of higher-return samples, effectively skewing the contribution of these samples during training. This enables the model to focus more on higher-return demonstrations. When p is set to 0, the RWDTP method reduces to pure behavior cloning, where all samples contribute equally, regardless of their return. Decision trees inherently offer greater explainability for policies. Figure 6 illustrates a decision tree from an ensemble used to form an expert-level policy in the CartPole environment. This specific tree shows that the agent's decisions are primarily based on the pole's angle: if the angle is negative, the agent is likely to push the cart left, and if positive, it mostly pushes the cart right. This results in an intuitive and interpretable weak policy. As more trees are added to the ensemble, the policy's strength and robustness increase. Feature importance analysis in figure 7 further enhances explainability, revealing that pole angle and angular velocity are more critical for stabilizing the pole than cart position and velocity. Similarly, in the pendulum environment, the policy equally prioritizes the cosine and sine components of the pendulum angle, giving substantial weight to all dimensions of the observation space in decision making. We have also visualized the decision boundaries learned by the decision tree policies. For CartPole, the decision boundary (whether to push the cart to the right) is visualized in figure 8a based on different pole angles and velocities, while keeping the cart's position and velocity fixed at (0,0). For the pendulum environment, we plot, in figure 8b , the predicted torque values over various pendulum angles and angular velocities, providing further insight into the model's action prediction surfaces. These visualizations further demonstrate the explainability of our approach across environments, highlighting how decision tree policies make control decisions. The offline dataset for the Racecar Gym experiments was obtained by implementing a controller tailored to the Austria racetrack. This controller receives the raceline to be tracked and determines the steering angle and motor command based on the current heading angle and lookahead points in the raceline. After training offline on a hundred episode demonstrations collected using this controller in the Austria racetrack, the agents undergo a zero-shot transfer test on various other racetracks. Both RCDTP and DT agents utilize only Lidar and velocity observations, excluding raceline or map information, to predict actions, which include steering angle and motor command. In this environment, agents are rewarded based on racetrack progress and are penalized for collisions, leading to episode termination.We conducted experiments using the hover-aviary environment in Pybullet drones. The target position is set to [0, 0, 1], with the reward based on the L4 norm distance between the drone positions and the target. The primary objective is to maintain a hovering position around z target = 1, with minimal deviations in the x and y dimensions from the starting state. Initially, we trained Proximal Policy Optimization (PPO [51] ) for 100,000 timesteps to obtain a 'suboptimal' agent and collected 100 episodes to form an offline demonstration dataset. Subsequently, we modified the z dimension of the dataset using the transformation z target -z so as to incorporate the goal height (z target = 1).After training RWDTP on this dataset with slightly modified observation space, we conducted simulations with z target values of 1, 0.8, and 0.6. The same RWDTP policy hovered the drone at each target height without any policy retraining, modifications to the reward function, or additional dataset collection.Table 7 presents the training times (on GPU) and inference times (on CPU) for 1 million gradient steps across various methods, as recorded in our experiments. For baselines, we compare CQL, Behavioral Cloning (BC), and different variants of EDAC with 10 and 50 critic networks. The experiments were conducted on a system running Ubuntu 22.04.3 LTS, equipped with a 13th Gen Intel Core i9-13900KF processor with 24 cores, an NVIDIA RTX 4090 GPU (24 GB), and 64 GB of DDR5 memory. In our attempt to improve the performance of decision tree policies (DTPs) on datasets without expert-level demonstrations, we conducted additional experiments using an IQL [42] critic to guide action selection. Unlike traditional methods, IQL allows training the Q function without requiring an actor to sample unseen actions, providing a distinct advantage in offline learning. Specifically, we integrated a perturbation layer with a mean of the predicted action and a standard deviation of 0.2 into the output of RCDTP and RWDTP methods. From these perturbed actions, we sampled N = 10 actions and selected the one with the highest Q value. This action selection (or policy extraction) method, based on sampling, has been employed before in [52, 53, 54] . The performance of these IQL-assisted DTPs is denoted by the suffix of +IQL in the table 8. Although the performance of 'plain' decision tree policies is on par with the conventional baselines in the non-expert regime, results of our experiments sometimes show performance improvements of up to 60% when using the IQL critic. This also highlights the better performance of decision tree policy prior in comparison to the diffusion BC prior used in IDQL [53] . To understand the performance gap between decision tree policies and their +IQL counterparts, we examine the distribution of actions learned by RWDTP and RWDTP+IQL in Hopper-Medium, where the gap is high, and Hopper-Medium-Replay, where the gap is lower. Figure 9 depicts the distribution of action distance (||a policy -a dataset || 2 ) evaluated over the observations in the dataset. Specifically, we examine the Hopper Medium and Medium Replay scenarios to illustrate the impact of potential overfitting on the performace. RWDTP's distribution of action distances from the dataset is highly skewed towards zero for Hopper-Medium compared to Hopper-Medium-Replay, indicating a lack of out-of-distribution generalization and potential overfitting within the dataset's distribution. As a result, when the IQL critic is used to guide the action selection, it shifts this action-distance distribution away from zero and also greatly improves performance. Feature importance analysis can be a crucial tool in robot learning, particularly for understanding which input features are most influential in the decision-making process of learned policies. By identifying key features, we can better assess the interpretability and robustness of these policies, which is essential for ensuring interoperability across different tasks and environments. In this additional study, we applied feature importance analysis to the Expert and Medium datasets in the Hopper and Walker2d environments to evaluate the behavior of the decision tree policies. Our analysis reveals that despite being trained on different performance datasets, RWDTP and RCDTP tend to prioritize similar features when predicting actions. For each environment, the two most important features identified by both methods (DTPs) in both datasets remain the same. The rankings of important features as identified by the two policies in the Hopper datasets are summarized in table 9 . This kind of consistency can be important for ensuring the interpretability of policies before deploying them to real-world robotic tasks.  
 
 Appendix:  


Decision: All reviewers are excited about this work. It dares to go against the current flow of using neural networks for low-dimensional tasks, and demonstrates strong merits of using decision trees. It achieves superior performance and yields explainable policies. This excitement is somewhat mitigated by a long list of technical questions by all reviewers, including recurring questions around reliance on expert demonstrations.

The authors provide additional detail and clarifications in their rebuttal. Acceptance would be on faith that these will be integrated into the paper as promised.

Some particular strengths of this paper:
- well written and easy to follow,
- strong performance on diverse robotic tasks,
- the method provides explainable policies,
- fast training times.

Questions mostly concerned technical details and were largely answered convincingly by the authors.

Reviews: review: title: title: Solving Offline Reinforcement Learning with Decision Tree Regression
paper_summary: paper_summary: This paper presents a method using Decision Tree Regression to learn optimal and explainable policies from offline data, showing strong performance on D4RL datasets, but raises concerns about potential overfitting and underfitting issues with non-expert data.
main_review: main_review: This paper proposes using Decision Tree Regression supervised learning to learn optimal and explainable policy using offline data. 
The approach shows superior performance on D4RL datasets for locomotion and manipulation, as well as other robotic tasks involving wheeled and flying robots. 

Strengths:
 - The paper is well-written and easy to follow.
 - The approach shows strong performance on D4RL datasets for a variety of robotic tasks.
 - The method provides explainable policies, which is beneficial for understanding decision-making.

Weaknesses
 - There are concerns about overfitting and underfitting when using offline data.
 - Questions remain about the effectiveness of the method when the offline data is not from expert trajectories.
 - The paper lacks clarity on the intuition behind using reward as the weight in the RWDTP method.
 - The approach's ability to avoid out-of-distribution actions is noted, but its limitations with non-expert data are uncertain.

Overall, the paper is well-written and easy to follow. The work is sound, and we can get an explainable policy using this method, but I am not sure about the overfitting and underfitting problems of offline data.
questions: questions: - In the RWDTP method, what is the intuition of using reward as the weighted for the lost?
 - The decision tree policies trained in this paper are based on avoiding out-of-distribution action distribution. 
 - Do you have problems with overfitting and underfitting when learning the optimal policy? What if the offline data collected is not from the expert trajectory?  Is it true that our performance is not competitive when we do not have an expert dataset?
limitations: limitations: 2
review_summary: review_summary: I would recommend weak reject and wait for the rebuttal from the author.
title: title: Interesting paper w/ opportunities for further investigation
paper_summary: paper_summary: This paper proposes methods to solve offline reinforcement learning problems via decision tree representations, incorporating return information either via conditioning (RCDTP) or by weighting the optimization objective (RWDTP) and running XG-BOOST. The methods are tested on a variety of continuous control offline RL task and display favorable performance, particularly in the expert demonstration regime, while significantly cutting down on training time.
main_review: main_review: - The paper is well-structured and easy to read. I appreciate the “simplicity” of the approach, departing from the more standard offline RL algorithmic design 
- The related work section should be extended to cover prior directions on using decision trees either directly for control or for post-hoc interpretability (e.g., [1]-[4])
- The baseline results on e.g. Gym-MuJoCo tasks are restricted to 1 hour training and often underperform the performance reported by the original authors (e.g. in Expert settings). It would be great to also report baselines results at convergence (w/ their associated run-time), which would increase transparency and allow others to make an informed decisions regarding trade-offs
- The F1 transfer experiments are very nice as they suggest the approach generalizing beyond the training distribution. It would be interesting to see how competitive the agent is, as it may learn only very simple heuristics. A trajectory overlay on a non-Austria track with a non-Austria expert policy and the RCDTP policy (+ video?) could be very informative. 
- Furthermore, analyzing the underlying decision boundaries similar to the autonomous driving setting in [4] would be useful (similar to the Cartpole analysis in the Appendix, which should be prominently referenced in the main text).
- While generalization for racing and drone flight are promising, a robustness analysis on the MuJoCo and/or Adroit tasks would further strengthen the results (e.g., noisy sensor readings or system delays).
- The reduction in performance on the non-expert datasets (e.g. Table 1) could indicate that clean, uni-modal data is needed. The discussion here should be extended.
- It would be nice to include training times for the BC experiments


**References:**

[1] Z. Ding, et al. “CDT: Cascading decision trees for explainable reinforcement learning.” arXiv, 2020.

[2] A. Silva, et al. “Optimization methods for interpretable differentiable decision trees applied to reinforcement learning.” PLMR, 2020.

[3] A. Pace, et al. “Poetree: Interpretable policy learning with adaptive decision trees.” arXiv 2022.

[4] T.-H. Wang, et al. “Measuring interpretability of neural policies of robots with disentangled tree representations.” CoRL, 2023.
questions: questions: - I might have missed this, but over how many seeds/re-runs were the standard deviation results computed?
- In the drone scenario, how was the goal height incorporated into the observations (agent height and difference to goal height)? Which initial conditions (agent height) does the expert data contain?
- At which dimensionality of both the state and action space would the method reach its limits?
- What are the values for the number of estimators and tree depth for each task?
- In Algorithm 1, does N refer to K?
limitations: limitations: 3
review_summary: review_summary: I lean towards accept as the method’s simplicity is a nice departure from the more conventional offline RL setting, with very fast training times that results in favorable policy performance. Quantitative and qualitative analysis could be strengthened for full transparency and higher impact.
title: title: good paper on a classical ML approach for offline RL datasets
paper_summary: paper_summary: Using gradient-boosted decision trees to solve offline RL tasks.
main_review: main_review: EDIT: Rebuttal acknowledged. I keep my accept recommendation as is.

In this paper, the authors train decision trees on low-dimensional offline RL tasks like the Mujoco locomotion tasks and Maze2d. They demonstrate surprisingly strong performance, often on-par with modern deep offline RL algorithms, especially on the datasets that are closer to expert-level. I think this paper is a good contribution to the conference. The audience will be interested in hearing about these results, and they call into question why we, as a community, are pouring so many resources into training deep models for low-dimensional offline RL. I have some concerns around strong claims, unclear assumptions, and the experiments section, which I highlight below.

Major Comments:
* Some strong claims are missing citations in the introduction. In particular, the lines "in many cases Behavior Cloning (BC) outperform offline RL algorithms" and "decision trees known for their superior performance in regression and classification tasks compared to neural networks".
* I think it would be good to discuss earlier on that decision trees are only applicable to low-dimensional offline RL tasks, i.e., tasks where the observation vectors are of a small dimensionality. A huge advantage of neural network function approximators is that they can learn directly from pixels as well, like in the original DQN paper. I know the authors discuss this in Section 5.5, but this feels too late to me; it's better to state the assumptions of your proposed approach earlier.
* A question I had throughout your experiments: Have you tried ignoring the reward signal and just doing pure behavior cloning with decision trees? I see that in Table 5, you are comparing RWDTP/RCDTP against BC, but I'm unclear if this is neural network BC or decision tree BC. To my understanding, there are two independent axes at play here: function approximator = {neural network, decision tree}; optimization strategy = {uses reward signal, pure behavior cloning}. Is this understanding correct?
* I would appreciate more discussion in the experiments section about _why_ RWDTP/RCDTP perform best when the data is more expert-level (stated in L198).
* Across the Experimental Results and Discussions sections, I felt that there is too wide a variety of domains, with some questions in the Discussions only being answered for 1 domain. In these cases, it would be better to provide a fuller set of results in the Experimental Results, and reference those in Discussions. For example, in Table 5 you only show BC results for Maze2d. Why was Maze2d chosen for this experiment? Why not include BC results for all domains in Table 1-3?
* Could you include a comparison to Decision Transformer for Table 1-3 as well?
* Eq 4: I got confused here. What happened to the $\tilde{R}_n^p$ from Eq 2? It seems like that got removed in Eq 4.
* Could you include some offline RL algorithms like CQL and EDAC to Table 6 (training and inference times) as well?
* Personally, I find the explainability aspect of decision trees to be the most exciting part of this work. Therefore, Fig 3 was very interesting to me. Would it be possible to also see this for a non-expert dataset, where decision trees didn't perform as well, like Hopper Medium? I wonder if we'd see the same qualitative results on that dataset.

Minor Comments:
* L26: "depending on the setting" It's more of a choice of which RL algorithm to use (which is decided by the researcher) rather than the problem setting (which is fixed & given to the researcher).
* You may be interested in checking out the paper "[When should we prefer Decision Transformers for Offline Reinforcement Learning?](https://arxiv.org/abs/2305.14550)"
* Eq 5: Shouldn't this be an average instead of a sum? If it's a sum, I don't understand why summing over actions suggested by each weak policy would make sense.
* L173: I'm unclear why RCDTP _needs_ $\gamma=1$. Can't we also do it with $\gamma<1$, by simply defining the return-to-go as a discounted sum?
* Figure 1: It would be better to show return on y-axis, and expert return as a dashed horizontal line
* L103: "the the" --> "the"
* L108: "XGBoost is faster" than what?
* L152: make sure to define $\gamma$ as the discount factor somewhere
* L152: "discounted sum of future returns" do you mean "rewards" instead of "returns"?
* Figure 2b: Wouldn't it be more natural to conduct this experiment with RCDTP instead of RWDTP? That way you can modulate by simply passing in a different target return. Then you don't have to "slightly modify the observation space" (L218).
questions: questions: See questions above, particulary in the Major Comments.
limitations: limitations: 3
review_summary: review_summary: Surprisingly strong performance by decision trees, which could generate good discussion at the conference.