Gradient Boosting Reinforcement Learning
Authors: Benjamin Fuhrer
Chen Tessler
Gal Dalal
Venue: ICML 2025 poster (2025)
Keywords: Gradient Boosting Trees
Reinforcement Learning
Machine Learning
Primary area: reinforcement_learning

Abstract: We present Gradient Boosting Reinforcement Learning (GBRL), a framework that adapts the strengths of Gradient Boosting Trees (GBT) to reinforcement learning (RL) tasks. While neural networks (NNs) have become the de facto choice for RL, they face significant challenges with structured and categorical features and tend to generalize poorly to out-of-distribution samples. These are challenges for which GBTs have traditionally excelled in supervised learning. However, GBT's application in RL has been limited. The design of traditional GBT libraries is optimized for static datasets with fixed labels, making them incompatible with RL's dynamic nature, where both state distributions and reward signals evolve during training. GBRL overcomes this limitation by continuously interleaving tree construction with environment interaction. Through extensive experiments, we demonstrate that GBRL outperforms NNs in domains with structured observations and categorical features, while maintaining competitive performance on standard continuous control benchmarks. Like its supervised learning counterpart, GBRL demonstrates superior robustness to out-of-distribution samples and better handles irregular state-action relationships.

Gradient Boosting Reinforcement Learning
Benjamin Fuhrer'
Chen Tessler? Gal Dalal 2
Abstract
complex feature extraction. Such structured observations closely resemble tabular datasets commonly encountered in supervised learning. Notable examples where structured observations are crucial include recommendation systems, healthcare diagnostics, digital advertising, fraud detection, weather prediction, and financial trading.
We present Gradient Boosting Reinforcement Learning (GBRL.), framework that adapts the strengths of gradient boosting trees (GBT) re- inforcement learning (RL) tasks. While neural networks (NNs) have become the de facto choice for RL, they face significant challenges with struc- tured and categorical features and tend to general- ize poorly to -iittiibuttttiino samples. These are challenges for which GBTs have traditionally excelled in supervised learning. However, GBT's application in RL has been limited. The design of traditional GBT libraries is optimized for static datasets with fixed labels, making them incompat- ible with RL's dynamic nature, where both state distributions and reward signals evolve during training. GBRL overcomes this limitation by con- tinuously interleaving tree construction with en- vironment interaction. Through extensive experi- ments, we demonstrate that GBRL outperforms NNs in domains with structured observations and categorical features while maintaining competi- tive performance on standard continuous control benchmarks. Like its supervised learning coun- terpart, GBRL demonstrates superior robustness to out-of-distribution samples and better handles irregular state-action relationships.
In these domains, key challenge lies in handling structured observations, where the importance of different components varies dynamically with the agent's s state and task. Neu- ral network (NN)-based solutions struggle with such data, requiring sophisticated architectures and preprocessing tech- niques to capture structural relationships (Zabërgja et al., 2024; Kadra et al., 2021; Arik & Pfister, 2021; Gorishniy et al., 2021; Hollmann et al., 2023). This often results in complex models that sacritice sample efficiency and gener- alization capability.
Gradient boosting trees (GBT) offer a promising alternative through their natural handling of structured data. Their suc- cess in supervised learning stems from an iterative ensemble- building process, in which each decision tree refines pre- dictions by partitioning the input space along individual features. This approach has made GBT frameworks such as XGBoost (Chen & Guestrin, 2016), LightGBM (Ke et al., 2017), and CatBoost (Prokhorenkova et al., 2018) integral in domains such as finance (Tian et al., 2020), healthcare (Wassan et al., 2022; Ma et al., 2022: Seto et al., 2022), and competitive data seience (Chen, 2023).
This approach creates an ensemble of piecewise constant functions that excel at capturing non-smooth patterns and abrupt transitions in data (Jeffares et al., 2024; Beyazit et al., 2023; Grinsztajn et al., 2022). These properties are par- ticularly valuable in RL, where state-action relationships often exhibit sudden changes, for instance, when transi- tioning between different phases of task or when certain state variables cross critical thresholds. Such discontinuous relat relationships common in structured RI. tasks, from in- ventory management where actions change discretely with stock levels, to game scenarios where strategic decisions shift abruptly based on state conditions.
1. Introduction
Many real-world decision- decision-making tasks involve structured observations, where data can be organized in tabular for- mat and follow predefined organizational patterns. Unlike unstructured data (such as images or raw sensor data), struc- tured observations often include heterogeneous features, both numerical and categorical. These features carry direct semantic meaning that can be used for prediction without
'NVIDIA, Tel-Aviv, Israel 2NVIDIA Research, Tel- Benjamin Fuhrer
Aviv. Israel. Conrepondence <bfuhrer@ mom.. Chen Tessler moiiaa.m@ Gal Dalal <gdalal @ nvidia.com>..
Despite these potential benefits, adapting GBT to RL presents significant challenges. Traditional GBT frame- works are designed for static datasets with stable feature distributions and predefined loss functions, where the focus
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Gradient Boosting Reinforcement Learning
is on optimizing batch training efficiency. This contrasts sharply with RL's dynamic nature, where state distributions evolve during training, data is collected iteratively, and re- wards are often delayed. While traditional GBT libraries can technically be adapted to RL using custom loss functions and incremental learning, our experiments with CatBoost and XGBoost (Section 4.3) show that this approach fails to scale due their supervised learning-oriente design.
graph NNs.
Despite their versatility, applying GBT in RL remains rela- tively less explored area. Several works have employed GBT as a function approximator within off- oif-policy RL methods, including its use in Q-learning (Abel et al., 2016) and in ban- dit settings to learn inverse propensity scores (London et al., 2023). Recently, Brukhim et al. (2022) proposed a boosting framework for RL where base policy class incrementally enhanced using linear combinations and nonlinear transfor- mations, forming 2-layer NN. However, these previous works have not demonstrated scalability and effectiveness in complex, high-dimensional RL. environments that require extensive interactions. In this work, we show how to adapt the framework of GBT to successfully solve large-scale RI. problems for the first time.
We address these challenges through Gradient Boosting Re- inforcement Learning (GBRL), a framework that bridges the gap between GBT's strengths and RL's unique requirements. Our contributions are:
New framework for GBT in RL: We establish GBT as viable function approximator for RL by devising popular algorithms-PPPO A2C, and AWR-with GBT backends. In popular environments, our framework demonstrates com- petitive performance against NNs while showing particular strengths on structured tasks.
Tabular Data. Previous work in RI. has focused predom- inantly on the use of NNs due to their ability to capture complex patterns in high-dimensional data. Techniques such as Q-learning and Actor-Critic methods have advanced significantly, demonstrating success in tasks involving raw sensory inputs like images, text, and audio. However, NNs that perform well on tabular data typically have very special- ized architectures (Katzir et al., 2021; Somepalli et al., 2021; Gorishniy et al., 2021; Arik & Pfister, 202 and are differ- ent from the standard multi-layer perceptrons that are often used in RL tasks and algorithms (Ota et al., 2024). Even with specialized architectures, GBT often performs equally or better on tabular datasets (Borisov et al., 2021; McBlfresh et al., 2023; Grinsztajn et al., 2022; Shwartz-Zir & Armon, 2022). GBT's success can be attributed to key difference in inductive biases compared to NNs. Jeffares et al. (2024) have shown that the kernel representations induced by GBTs are bounded and often behave more predictably on irregular out-of -f--iiiibiitttioon inputs. In contrast, NN-tangent ker- nels can be unbounded and vary significantly on test points far from the training distribution.
We demonstrate that GBT's in- herent strengths translate effectively to RL tasks. Through extensive dedicated experiments, we show superior robust- ness to -iittr-diittibttion scenarios, resilience to spurious correlations, and better handling of state-space perturbations -critical advantages over NNs for real-world applications.
2. Improved robustness:
3. Practical implementation: We provide a CUDA- accelerated (NVIDIA, 2025) implementation that seam- lessly integrates with existing RL libraries like Stable- baselines3 (Raffin et al., 2021)! Our shared Actor Critic design achieves late-of-thhear performance while signif- icantly reducing both memory and computational require- ments on modern hardware.
2. Related Work
GBT Beyond Supervised Learning. Recent advances have extended the capabilities of GBT beyond traditional regression and classification. In ranking problems, GBT has been used to directly optimize ranking metrics (Lyzhin et al., 2023), as demonstrated by frameworks such as Stochasti- cRank (Ustimenko & Prokhorenkova, 2020). Furthermore, GBT offers probabilistic predictions through frameworks such as NGBoost (Duan et al., 2020), allowing uncertainty quantification (Malinin et al., 2021). The connection be- tween GBT and Gaussian processes (Ustimenko et al., 2023; Sigrist, 2022) offers further possibilities for uncertainty- aware modeling. Recently, Ivanov & Prokhorenkova (2021) modeled graph-structured data by combining GBT with
Policy Optimization through Functional Gradient As- cent. Kersting & Driessens (2008) introduced Non- Parametric Policy Gradients (NPPG). By combining the policy gradient theorem (Sutton et al., 1999) with func- tional gradient ascent, NPPG directly optimizes the policy. Although NPPG sets the foundation for combining GBT with RL, it is limited to discrete action spaces and policies represented by the Gibbs distribution. GBRL extends the ideas laid in NPPG to general Actor-Critic algorithms and generalizes to any policy representation and objective.
'The GBRL core library is available at https:// withhub.com www.///aao//////ooooo/owooooaeee 2 'Actor-Critic implementations integrated within Stable- baselines3, are available at ht atsps:/ :/ //gg wig.thub.co//Neee /Vlaas// gbrl_sb3.
3. RL Preliminaries
A fully observable infinite-horizon MDP is characterized by the tuple (S,A, P, R). At each step, the agent observes
2

---

Deep RL with NNs faces significant challenges when deal- ing with structured observations containing both numerical and categorical features. Similarly to tabular datasets in supervised learning. these observations consist of heteroge- neous features representing fundamentally different types of information, such numerical measurements alongside categorical attributes describing system states task con- ditions. These features create highly irregular patterns, as they may independently affect the environment based on Gradient Boosting Reinforcement Learning
F =
D = {st, V Ji
Fo, hg
h1
hs
F1
F6
RL Objective (PPO/A2C/AWRI...
Policy
Value
Environment
action
feedback
Figure 1. The GBRL framework. The actor's policy and critic's value function are parameterized by the tree ensemble Fk. For example, (s) (s), V(s) for Gaussian policy. Starting from Fu, at each training iteration, k. GBRL collects rollout and computes the gradient F= (TTR) with respect to the current ensemble. This gradient is then used to fit the next tree. Adding the tree to the ensemble updates (s) (s) (s), where is the learning rate.
a state 8 E S and samples an action a A from policy (s,, a). The action causes the system transition to new state s' based on the transition probabilities P(s' | a), and produce reward is to maximize the expected discounted reward ((x)
their context, scale, or statistical properties. NNs struggle to model such patterns effectively (Beyazit al., 2023; Jef- fares et al., 2024), and their performance heavily depends preprocessing methods (Hancock & Khoshgoftaar, 2020; Gorishniy et al., 2022). In contrast, GBT naturally handles irregular patterns and structured feature spaces efficiently, To leverage this strength, we present GBRL, an adaptation of GBT to RL tasks with such representations that frequently arise in real-world problems.
R(s, a). The agent's objective
E re], with discount factor y E [0, 1).
Iz],
The state-action value function (s,
E, E 7¹ rt 8o 8, a] estimates the ex- pected returns of performing action a in state s and then acting according
n. Additionally, the value function (s) = E,, 12re y'1, |so s). predicts the expected return starting from state s and acting according to æ. Fi- nally, the advantage function (s, a) == Qx a) - (s),,, indicates the expected relative benefit of performing action a over acting according to π.
4.1. GBT as Functional Gradient Descent
GBT (Friedman, 2001) combines decision tree ensembles with functional gradient descent to learn complex nonlinear functions (Mason al., 1999). Given a loss function L and dataset, D {(x,, (is4,ys)) ),N where x; E R and y, R?, GBT aims to learn a function F, mapping inputs to outputs y. Through functional gradient descent, GBT minimizes the expected loss Exx.y[(yy F(x)) with respect to the learned function F. Unlike parametric gradient descent, which updates parameters, functional gradient descent updates a learned function directly.
Actor-Critic methods are a common approach in RL. They simultaneously learn policy and value function to im- prove the efficiency and stability of training. For concise- ness, in the body of the paper we focus on Proximal Policy Optimization (PPO) (Schulman et al., 2017) which jointly trains an actor and critic. In the supplementary material, we also showcase the implementation and results for A2C (Mnih al., 2016) and AWR (Peng al., 2019).
Starting with an initial function Fo. At each iteration k. GBT refines Fg. by taking step in the negative direction of the gradient 9ı: := V p. L(y, (x)). The negative gradient is projected onto the space of possible trees h E H. by constructing new binary decision tree that minimizes the following objective:
4. Gradient Boosting Reinforcement Learning
hy = arg min || - gr - 1(x) RER
(1)
Through this process, GBT minimizes the expected loss, resulting in the additive model:
K
Fx Fx (x) = Fo+
Σ ch+(x):
(2)
Am
3

---

Gradient Boosting Reinforcement Learning
Library
Incremental Learning (sec)
Standard Learning (sec)
Frames per Second
10a
CatBoost
36.17 2.77
1.15 1 0.03
XGBoost
67.11 + 0.70
2.26 + 3.70
GBRI.
5.47 + 1.19
N/A
10a
Table 1. Incremental Learning Speed in GBT Libraries. Train- ing times (seconds) 1000 boosting iterations on random batches of 128 samples with 20 imput features and 3 targets. While Cat- Boost and XGBoost train efficiently in standard mode using the same batch for all iterations, their training time increases sigmif- icantly in incremental learning. where each batch receives only boosting iteration. GBRL, designed explicitly for incremental updates, remains much faster.
8000
10000
14000
16000
Boosting Iterations
GBRL
XGBoost
CatBoost
Figure 2. GBT library comparison, Cartpole. CatBoost and XGBoost intractable in RL. CatBoost's lack of GPU support
for custom losses leads to low FPS and early termination.
where, is the learning rate controlling the step size in each iteration.
Epiiondic Reward
60
Franess per Second
100-
0.75
Allocated
CIU Menory
40
75
0.5-
While standard GBT frameworks excel at handling struc- tured data supervised tasks, they require a tailored solu- tion for RL's iterative, dynamic, and multi-objective nature.
0.25-
20
5
10
2.5 7.5
100200
Timestops [M]
Timesteps [M]
Boosting
Iteaatinns [K]
Shared AC
Soparate
4.2. The GBRL Framework
In the GBRL framework (Figure 1) we adapt GBTs to handle the challenges of RL, To do so we reinterpret Fx as the parameterization of the policy and value function, where the observed state S is the input to the ensemble.
Figure 3. Shared Actor-Critic, MiniGrid. The shared tree struc- ture significantly increases efficiency, without impacting the score. Aggregated results over three tasks are shown here, full per-task curves are available in Appendix (Figure 20)
Specifically, at each iteration k; we compute the policy gradient (Sutton et al., 1999). As opposed to the original formulation, where the gradient is calculated with respect to parameter space, here we calculate function space. Consequently, we express the policy gradient as:
with most Actor-Critic algorithms and implemented within existing RL libraries. In contrast, traditional GBT frame- works are designed for offline, static datasets and require significant customizations and workarounds to function in RL.
(TFF) = Ear, [V, log nF (a|s) a)],
Shared Actor-Criti Architecture. Many RL algorithms operate with shared Actor-Critie structure. This structure helps avoid overfitting and improves overall convergence speeds (Henderson et al., 2018; Andrychowicz et al., 2020), We adopt this approach in GBRL such that each leaf pro- duces two predictions. GBRL predicts both the policy (dis- tribution over actions) and the value estimate. The internal structure of the tree shared, providing single feature representation for both objectives and significantly reduces memory and computational bottlenecks. Similar to the stan- dard practice, support and use separate learning rates for the policy and value. This approach enables solving two distinct objectives within single shared structure.
where F3 is the current ensemble-based parametrization.
Then, the policy gradient is projected onto a new tree and added to the ensemble:
hy = arg min J(TA.) h(s)]l
(4)
NEH
The result is application of GBT functional gradient optimizer, updating both actor and critic incrementally:
(s Fo
Σ chu(s) R Fo Nr J(Tr)) (5)
Gradient-based Approach. As seen in Equation (5), GBRL is an online learning gradient-based optimizer. In each step, new tree is constructed to minimize the sam- pled loss. This learning scheme is identical to the common policy gradient methods (Mnih et al., 2016). By leveraging optimization frameworks for gradient computation, such as PyToorch (Paszke et al., 2019), GBRL, can be integrated
4.3. Comparison to Traditional GBT Libraries
Before diving into the comparisons with NNs, Section 5, we first evaluated our design choices against traditional GBT libraries. While these libraries can be adapted for RL using custom loss functions and incremental learning, experiments with CatBoost and XGBoost reveal scalability

---

Gradient Boosting Reinforcement Learning
issues inherent their supervised learning design. Table shows these frameworks significantly slower than GBRL, when used for incremental learning. Incremental learning is a fundamental property of RL, as the policy is continually changing. Moreover, when training on CartPole (Figure 2), larger ensemble sizes lead to significant drops in training throughput. In contrast, our shared Actor-Critic architecture halves memory usage and doubles throughput without com- promising policy quality (Figure 3). Overall, GBRL offers purpose-buil solution that maintains GBT's advantages while efficiently handling RI RI-speciiic needs.
ing goal-oriented tasks with discrete object interactions.
Results. We report the cumulative non-discounted reward, averaged across the last 100 episodes in Figure 4. For a sim-
ple visual comparison between GBRL and NNs, we report
the normalized score: Fnorm
F
Faio GBN.)
1.0-
Normalized Perioedeeraaaa
0.8-
0.6
0.4
5. Experiments and Results
0.2
Our experiments aim to answer two core questions:
0.0
1. GBT as an RL function approximator: Can GBT-based
MiniGrid
Football
Continuous
Atari-ram
Control
AC algorithms effectively solve complex high-dimensional RL tasks? And how do they compare with NNs?
Domains
GBRL
NN
2. Advantages of GBT: Building on the GBT's success in irregular and tabular data, does its inductive bias offer similar robustness benefits in RL specifically for out-of- distribution states, noisy inputs, and spurious correlations?
Figure 4. GBRL vs NN in standard environments (PPO), Ag- gregated mean and standard deviation of the normalized average reward for the final 100 episodes.
For our experiments, we implemented GBT-based version of PPO within Stable Baselines3 (Raffin et al., 2021). Where available, we use standard hyperparameters. environment- specific, and normalization wrappers according to RL Base- lines3 Zoo (Raffin, 2020); otherwise, we optimize the hyper- parameters for specific environments, For each experiment, report the aggregated performance across five random seeds. Our training setup consists of single NVIDIA V100 GPU. We refer the reader to the supplementary material for additional technical details, such as hyperparameters, implementation, and environment details (Appendix C), full results (Appendix D) and full learning curves (Appendix E) for all evaluated algorithms.
Decision trees, at their core, are if-else structures that excel in handling numerical and categorical data. In MiniGrid, which provides such environments, GBRL significantly out- performed NN. For example, in tasks like PutNear, Four- Rooms, and Fetch, GBRL. consistently achieved higher re- wards, highlighting its ability to exploit the structured nature of the environment effectively. This performance demon- strates that GBRL is particularly well-suited for problems where data can be neatly partitioned, aligning with GBT's inherent design strengths. This suggests that structured en- vironments represent an ideal use case for GBRL.
An additional domain characterized by structured representa- tions is the Football domain, where we used a shaped reward. Here, features such as player positions and ball locations were designed to represent identifiable, interpretable infor- mation. In this domain, GBRL outperformed NN across most scenarios and exhibited equivalent performance in the rest. The structured nature of the Football domain aligns well with the strengths of GBT, effectively partitioning the feature space. Interestingly, while both Football and MiniGrid are structured, the Football domain is not sparse. Hence, GBRL's strong performance in Football underscores its robustness and capacity to generalize well in structured, high-dimensional environments.
5.1. Standard Environments
Experimental Setup. We evaluated GBRL against NNs across three categories of RL benchmarks. First, we tested classic RL tasks using Classic-Control and Box2D envi- ronments from Gymnasium (Towers et al., 2024). These provided a baseline for comparison on standard benchmarks. We then evaluated both methods on more complex envi- ronments with high-dimensional vectorized representations: the Football domain (Kurach et al., 2020), where we em- ployed the built-in 'Checkpoints' shaped reward, and the Atari RAM domain (Bellemare et al., 2013). Finally, we assessed performance on categorical environments, specif- ically targeting the MiniGrid domain (Chevalier-Be Boisverrt et al., 2023)- a setting where GBTs have traditionally ex- celled in supervised learning. MiniGrid provides an ideal test environment through its 2D grid-work world structure featur-
Unlike Football, the Atari RAM domain offers unstructured, high-dimensional representations by exposing raw system memory states. These flattened views lack explicit seman- tic structure, making them challenging for GBRL, which relies on single-feature splits. As a result, GBRL underper- formed compared to NN in most Atari RAM tasks. This
5

---

Gradient Boosting Reinforcement Learning
result highlights important limitation of decision-tree- based approaches. They struggle to model implicit feature interactions, inherent characteristic of unstructured data.
=bcc
by +c=
2x
h = cy
Reward
1,00
0.75
1.00
0.75
1.00
0.50
0.30
0.25
0.25
0.25
0.00
0.00
0.00
Finally, classic control tasks are simpler environments that rely on low-dimensional state spaces. In these tasks, GBRL demonstrated performance comparable to that of the NN counterpart. This suggests that decision tree-based models can match NNs in solving tasks where feature interactions are limited and the optimization landscape is smooth.
Length
0 15 20
30 Timesteps [M]
10
10
Timesteps [M]
Timesteps [M]
GBRL
NN
Overall, these comparisons indicate that GBT can serve strong function approximator in RL, matching or surpassing NN's performance in certain environments.
Figure 5. Signal dilution, variable isolation task. Mean and standard deviation of the average episodic reward during train- ing. GBRL, was trained for 15M training steps and NN for 30M. Episodes are terminated after 50 steps if not solved.
5.2. Challenge Environments
Having established GBRL's general capabilities, we now examine scenarios designed to test inherent properties of decision trees. These experiments focus on challenging conditions- sparse rewards, misleading correlations, and input perturbations NNs typically struggle.
icy throughout the process. GBTs naturally segment the decision space into explicit rule-based splits, preventing sig- nal dilution while correctly preserving the rare but crucial reward signals. Consequently, we see that GBRL can pre- serve and exploit rare but critical reward signals-evidence evidence that Cisse methods are inherently less prone to signal dilution.
5.2.1. SIGNAL DILUTION
Signal dilution occurs when rare events, such as sparse rewards, are statistically overwhelmed by more common non-event data during training (Shyalika et al., 2024; Zhao et al., 80; He& & Cheng, 2021; Tessler et al., 2017). In NNs with fixed architectures, shared weights inherently prioritize patterns from dominant classes frequent signals, causing rare events to be "averaged out" or under-represented in learned representations. This phenomenon is particularly problematic in RL and classification tasks with imbalanced datasets, where the model fails to retain or react to critical but infrequent signals.
5.2.2. SPURIOUS CORRELATIONS
Experimental Setup. The agent is placed in a grid envi- ronment with three balls (red, green, and blue), and red box. The ball locations are randomized. The agent observes an objective specifying which ball to pick up. We exper- iment with three box- hox-placement configurations: adjacent to the target ball, adjacent to one of the non-target balls, and without box in the room. This task tests whether the policy overfits spuriously correlated features. We then analyze the results using SHAP (Lundberg & Lee, 2017), a method based on Shapley values (ARROW et al., 1953) that measures the contribution of each feature to the final prediction of a model.
Experimental Setup. We compare GBRL with NNs in a linear equation variable isolation environment with sparse re- wards, inspired by (Poesia et al., 2021). The agent observes the coefficients (a, b, c) of a linear equation ax + b c, and needs to isolate receiving a large reward only upon com- pletion. Actions are multi-discrete involving the selection of an operation, a digit E [0, 9], and its sign. The optimal policy solves the simple task in steps. We also evaluate two more complex variations with an additional variable y, where the optimal policy requires 3-4 steps. For these, we introduce intermediate goals with small rewards guide the agent.
Results. GBRL consistently outperforms NNs in all see- narios (Figure 6). demonstrating its ability to ignore irrel- evant distractions, such as the box, and reliably reach the target regardless of its placement.
To understand GBRL's performance, we analyzed the SHAP values for the agent's surroundings and the mission feature when predicting the logit for picking up the target object (Figure 7). GBRL assigns the strongest positive SHAP value to the target object, followed by the mission string, aligning perfectly with the goal. Non-target objects receive the most negative SHAP values, suppressing distractions and reinforcing goal-directed behavior. In contrast, NNs assign negative SHAP values to the mission feature and rely heavily on the grid surrounding the target object. This indicates a reliance on spurious correlations and a lack of
Results. While simple for humans, these tasks remain challenging for NNs due to signal dilution (Figure 5). Al- though they can make some progress in the simple task (ar + b = c), they fail when the equation complexity in- creases, even when given intermediate rewards. In contrast, GBRL converges to and retains a stable (and optimal) pol-
6

---

Gradient Boosting Reinforcement Learning
No-Box
1.00
Mission: get red ball Action: pickup object
0.75
0:30
0.385
Episona Reward
0.00
1.0
Training
Adjacent
to Target
1
SHAP Value
0,5
0.5
Non- Non-Targe Target
Adjacent
0.0
0.0
0.50
-0.5
-0.5
0:25
0.00
No-Box
Adjacent to Target Non-Target Adjacent
1.0
1.0
Evaluation
Mission
GBRL
NN
(a) GBRL SHAP.
Mission:
get red ball
Figure Spurious correlations, MiniGrid. Mean and standard deviation of the episodic reward. Plots outlined in green in- distribution, while plots outlined in red are out-of-distribution.
Action:
pickup object
1.0
SHAP Value
0.5
goal-oriented strategies.
0.0
0.0
These findings underscore GBRL's ability to focus on task- relevant features and ignore deceptive cues, reinforcing our hypothesis that GBT's inductive bias provides robustness against spurious correlations. Next, consider how this robustness extends other forms of state perturbations.
-0.5
-0.5
1.0
1.0 Mission
(b) NN SHAP
5.2.3. ROBUSTNESS TO STATE PERTURBATIONS
Figure 7. Spurious correlations revealed by feature importance in MiniGrid. The visualization shows normalized SHAP values the concluding action a successful evaluation run, revealing the magnitude and direction of each feature's impact model predictions. this scenario, target (red ball), and both models converged to successful policy. GBRL prioritizes the mission, ignores the red box, and contrasts the blue ball. In contrast, the NN focuses the cells surrounding the target, while ignoring the blue ball and contrasting the box and the mission.
When encountering an out-of-distributiin (OOD) state, a GBT model will always output a result within the same range seen in training. In contrast, NNs may extrapolate and exhibit unexpected behavior (Xu et al., 2021: Meinke & Hein, 2020; Ulmer & Cinå, 2021). We consider three tasks. First, robustness to irrelevant information. Second, robustness to random noise. And finally, robustness to miss- ing features. These experiments aim to test the inherent robustness properties GBT's when encountering different types noise during training.
box was positioned next to the
and efficiently solves the task, regardless of the additional irrelevant object placed in the scene.
Irrelevant Information. We consider a similar fetch ex- periment before (Section 5.2.2). Now, a new object is random location. This object is
placed in the not part of the mission and only interferes with navigation. The interfering object differs from the task objects in shape, color, or both.
Random Noise. In the classic-control environments (Sec- tion 5.1), train agents in noiseless setting. Then, to evaluate the robustness of the resulting policies, we add random zero-centered Gaussian noise to the input states as defined by: st = St +c|sa
Figure 8 illustrates how introducing a distractor impacts policy performance. When the environment is free dis- tracting objects, NNs able solve the task, converging stable and ideal solution. Surprisingly, although the purple-box object shares no features with the task objects (blue, green, and red balls), NNs exhibit drop in perfor- mance but are able to solve the task in a subset of seeds. This performance degradation increases when the confusing object shares more features with the goal objects, such as purple-ball and red-box. In contrast, GBRL successfully
The results show that these characteristics translate to practi- cal outcomes (Figure 9). classic-control environments. trained GBT-based policy is much more robust small vari- ations in the state space compared to NNs. We attribute this performance to how GBTs partition the input space. This partitioning occurs on hard thresholds that are split accord- ing to single features. Although small perturbations may sufficient to completely change decision outcomes in NNs (Lütjens et al., 2020; Carlini & Wagner, 2017), GBRL

---

Gradient Boosting Reinforcement Learning
Randomly Placed
Counterattack Hard
11 vs 11 Lazy Opponent
No Box
Purple Ball
Reward
1.00
1.5
1.0
0.00
Training
0.0
Randomly Placed
Randomly Placed Purple
Red Box
2.0
Reward
1.00
1.5
Dme
0.00
0.5
15 Timesteps [M]
20
10
15
20
Timesteps [M]
Full
Drop
Full
Drop
GBRL
NN
Evaluation
GBRL
NN
Figure Robustness to irrelevant objects, MiniGrid. Mean and standard deviation of the average episodic reward during training. GBRL was trained for 5M training steps, and NN for 20M.
Figure 10. Out-of-distribation evaluation, foothall. Mcan and standard deviation of episodic reward across 100 evaluation episodes. Plots outlined in green are in-distribution, while plots outlined in red are out-of-distributton..
MountainCar
Continuous-vo
MountainCar-
CariPole-vl
Roward
100
50-
-225
100
500
0-
Egisodic Reward
500
-50-
175-
-100
400
0.05 0.150.2
0..50.0
0.050. 10.15..2
300-
Variance
Variance
Variance
200
GBRL
NN
Figure 9. Robustness to input perturbations, classic-control. Models were trained without perturbations then evaluated on noisy observations. At each timestep. the observation was per- turbed following Si = St + Rt where e N(,, var). We report the mean and standard deviation of episodic reward across 100 evaluation episodes each Variance E [0, 0.2].
0.25
0.50
1:0000000
0.25
1130
Training Timesteps
x10
Training Timesteps
x10F
Learning Rate
0.0001
0.001
0.01
0.1
0.0005
0.005
0.05
Figure II. Ablation: Learning rate variation across policy (left) and value function (right) components. Mean and standard devi- ation of the average episodic reward during training. Fach subplot shows the effect of varying the learning rate for one component (policy or value function) while keeping the other fixed.
exhibits robustness to random additive noise.
Missing Features. We modify scenarios in the football domain (Section 5.1) to support the random drop of player from the opposing team. Policies are trained on both the original task and modified variant. We then evaluate model robustness by testing how a policy trained against random players performs when faced with N + 1, or how policy trained against full team behaves when random opponent is removed.
parameter at time while keeping the others fixed. For each experiment, we trained an agent in the CartPole-vl environment for 1M timesteps using GBT-based PPO.
Learning Rate Variation Across Policy and Value Func- tion Components. We investigated the impact of the learn- ing rate for both the policy and value function components. In these experiments, we held one learning rate constant while varying the other. Consistent with observations in NN training, we found that excessively high learning rates destabilized the learning process, whereas overly low rates led significantly slower convergence (as illustrated in Figure 11).
The results show that GBRL outperforms NNs in OOD tasks (Figure 10), particularly in the 11 vs 11 academy scenario. Notably, GBRL maintains performance when evaluated against opponents can solve vs. 11 even when trained on 10. In contrast, NNs struggle in all OOD cases. This suggests GBRL better isolates features that are essential solving the task.
6. Ablation Study
Limitations of Tree Depth. We varied the maximum depth the decision trees built at each training iteration and evaluated their impact on learning dynamics. Figure 12 demonstrates that deeper trees yield more accurate gradi-
In this section, we examine the sensitivity of GBRL to key hyperparameters. We systematically changed one hyper-
8

---

Gradient Boosting Reinforcement Learning
50
Overall, these ablations highlight that GBRL's performance is sensitive training frequency, tree depth, and batch size-each of which governs the trade-off between con- vergence speed and stability. While this section focused on general hyperparameters, we further investigate how these trade-off's manifest across different RL algorithms Ap- pendix B.
Episodic Reward
500-
Runtine [min]
40-
400-
30-
300-
200-
20
100-
10
0.00
0.25
01.00
0.75
1.00
Training Timestops
X10P
Max Depth
10
7. Limitations and Future Directions
GBRL exhibits competitive performance and surprising ro- bustness capabilities. However, despite these benefits, we highlight two important limitations. (i) Continuous gener- ation of trees. As the policy improves through numerous updates, the size of the ensemble increases. This unbounded growth has implications for memory usage, computational efficiency, and the feasibility of online real-time adaptation. Future work may leverage tree redundancy to produce more efficient and compact policy. For example, strategies for tree pruning, ensemble compression, or dynamically managing ensemble size. (ii) Off--policy continuous con- trol. Methods like DDPG (Lillicrap et al., 2019) and SAC (Haarnoja al., 2018) update the policy by differentiating the Q-estimator. As GBTs not differentiable, new solu- tions are needed to incorporate them into these algorithms.
Figure 12. Ablation: Effect of tree depth on convergence. We varied the maximum depth the decision trees used in the gradient-boosted models and evaluated their impact on learning dynamics. The plot shows the mean and standard deviation the average episodic reward over time and the total runtime.
Episodic Reward
500-
400-
300
200
100
0.25
0.50
0.75
Training Timesteps
x10"
Batch Size
8. Conclusion
128
512
2048
Historically, RL practitioners have relied on tabular, linear, and NN-based function approximators, While GBT evolved widely successful tool in supervised learning, it has been absent from this toolbox. In this work, introduced method for effectively integrating GBT into RL. We demon- strated domains where GBT excels compared to NNs, in addition to analyzing its various inherent characteristics.
Figure 13. Ablation: Batch size impact on convergence. Mean and standard deviation of the average episodic reward during train- ing. We fix the rollout length to 2048 and vary the batch size. Smaller batches improve early learning but suffer from instability, while larger batches stabilize training at the cost of slower conver- gence.
We observe that the ideal method depends on the task char- acteristics. Tabular and linear approaches are suitable for small state spaces simple mappings, while NNs excel when tackling complex relationships in unstructured data. In comparison, GBT thrives in complex, structured envi- ronments. In such cases, GBRL's advantage is highlighted, reflecting GBTs known benefits from supervised learning.
ent approximations, leading to faster convergence, but also incur higher computational cost per training step. In the paper, selected maximum tree depth of 4, balancing convergence speed and wall-clock time.
Batch Size Impact on Convergence. We analyzed the impact of the batch size on performance. In Figure 13, we observe that batch size significantly impacts convergence. Specifically, smaller batches result GBRL building more trees per rollout, improving adaptability. However, smaller batches also lead noisier gradient estimates as a result of limited samples per constructed tree. Conversely, larger batches stabilize training by reducing variance through av- eraging within leaves, as more samples are utilized to con- struct each tree but build fewer trees per rollout. Hence, both excessively small and large batch sizes negatively impact performance.
A crucial component GBRL is our efficient adaptation of GBT for Actor-Critic methods, which allows the simultane- ous optimization of distinet objectives. We optimized this approach for large-scale ensembles using GPU acceleration (CUDA). Furthermore, GBRL integrates seamlessly with existing RL libraries, promoting ease of use and adoption.
GBRL is a step toward well-suited solutions for real-world tasks with structured data.

---

Gradient Boosting Reinforcement Learning
Impact Statement
Borisov, V., Leemann, T., Seßler, K., Haug. J., Pawelczyk, M., and Kasneci, G. Deep neural networks and tabular data: A survey. JEEE Transactions Neural Networks and Learning Systems, 35:7499-75191 2021. URL. https: /apps///ppi http://pp..saaaa// ccshol caccolar.orr/ CorpusID:23635389
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
References
Brukhim, N., Hazan, E., and Singh, K. boosting approach
reinforcement learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- ume 35, pp. 33806-33817. Curran Associates, Inc., 2022. URL https://prroceed.gg..eurrr peper._fllee/ppper//222//// (af5864f00755c41144477777cca. Paper-Conference.ppee
Abel, D., Agarwal, A., Diaz, F., Krishnamurthy, A., and Schapire, R. E. Exploratory gradient boosting for re- inforcement learning in complex domains, 2016. URL ftttp::/aaxxiv.oo abss///r/////////ooo////sss/sss/ abs 1/111.0000111
Andrychowicz, M., Raichuk, A., Stanczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., Gelly, S., and Bachem, O. What matters in on-policy reinforcement leanninn a large-scale empirical study, 2020. URL attps://arxiv.oxx aariv.orgg/ 2tte/2066.5550
Carlini, N. and Wagner. D. Adversarial examples are not easily detected: Bypassing ten detection meth- ods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSee '17, pp. 3-14, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450352024. doi: 10. 10.11455/122877:31444444 URL https: http::://ddo.rrr 10.145/312572.334044
Arik, S. O. and Pfister, T. Tabnet: Attentive inter- pretable tabular learning. Proceedings of the AAAI Conference on Artificial Intelligence, (5(8):6679-6687 May 2021. doi: 10. 16.9/a9aiii v35i8. 16826. URL
:ttps:///os..aaa..org article/viiw/11666.
Chen, T. Machine learning challenge winning solutions, 2023. ht.ps:://github.com// c/ htxpboost//////oooooooooooco..
ARROW, K. J., BARANKIN, E. W., BLACKWELL, D.,
BOTT, R., DALKEY, N., DRESHER, M., GALE, D.,
hxte:mmasterr/mmmottutttteeeeeee challengee-iinnnnng--ouuttttrr
GILLIES, D. B., GLICKSBERG, GROSS, O., KAR-
LIN, S., KUHN, H. W. MAYBERRY, P., MIL-
Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Dis- covery and Data Mining, KDD '16. ACM, August 2016. doi: 02.11455729996722299777 URL http: / //Bkkbbl.crg//01115/333922222223
NOR, J. W., MOTZKIN, T. S., NEUMANN, J. V.,
RAIFFA, H., SHAPLEY, L. S., SHIFFMAN, M.. STEW- ART, F. M., THOMPSON, G. L., and THRALL, R. M. Contributions to the Theory of Games (AM-28), Vol- une II. Princeton University Press, 1953. ISBN 9780691079356. URL hett::/www.jjttnn/ httpbl/e...cttbbxxee1
Chevalier-Boisvert, M., Dai, B., Towers, M., R., Willems, L., Lahlou, S., Pal, S., Castro, P. S., and Terry, J. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36. pp. 73383-73394. Curran Associates, Inc., 2023. URL. paper ssper.fille//ppeer///////Le e6211198466682221222884444444999 Paper-lattastts.anddeeecccc
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation plat- form for general agents. Journal of Artificial Intelligence Research, 47:253-279, June 2013. ISSN 1076-9757. doi: 10. 1..16//aairr URL http::ttt///ooeo /x..oi.org//////ooooooooooooo./. 10.1 / 10.1613/jaix.3911. ja1r.391..
Beyazit, E., Kozaczuk, J., Li, B., Wallace, V., and Fadlallah, B. An inductive bias for tabular deep learning. In Oh. A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. URL 88116BffccB444cBBBc2221992 Paper-Conference.pdff
Curran Associates, Inc., 2023. Sapert.flees/pppe////2/////////
Duan, T., Anand, A., Ding, D. Y., Thai, K. K., Basu, S.. Ng. A., and Schuler, A. NGBoost: Natural gradient boosting for probabilistic prediction. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning,
10

---

Gradient Boosting Reinforcement Learning
volume 119 of Proceedings of Machine Learning Research, pp. 2690-2700. PMLR, 13-18 Jul 2020. URL htttss::/ppooceeeee httat::://ooreennggmm...eesss duan20a.html
020-00305-w. URL https:
http:///././ot////oooooooo////// /ttps/////b..ro//..11188. s -40537-020-00305--
He. and Cheng, M. X. Weighting methods for rare event identification from imbalanced datasets.
Friedman, J. Greedy function approximation: A gradient
Frontiers in big Data, 4:715320, 2021. doi:
boosting machine. The Annals of Statistics, 29(5):1189
10.389//fdaa.2222.1155500
URL https://
1232, 2001. doi: 10. (0.12114/o0//0012003333
//doi.org/10.12/1
axticles/10.383 http:///daaaaaa..aa.... Ser.clees/...B8//.ac...22..1....
Gorishniy, Y., Rubachev, I., Khrulkov, V., and Babenko, A. Revisiting deep learning models for tabular data. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 18932-18943. Curran Associates, itci URL attp:://proceedis..eeee papo...ileesppppe:22011// Paper.pdf.
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep reinforcement learning that matters. Proceedings of the AAAI Conference Ar- tificial Inielligence, 32(1), Apr. 2018. doi: 10.1609/ aaai. mmai.v32il..1 132.1.1161.1.. URL https:/ //t.ee///////////rrricc/////////
2021.
Hollmann, N.. Müller, S., Eggensperger, K., and Hut- ter, F. Tabpfn: A transformer that solves small tab- ular classification problems in second, 2023. URL https://arxive
Gorishniy, Y., Rubachev, I., and Babenko, A. On embeddings for numerical features in tabular deep learning. In Koyeja, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24991-25004. Curran Associates, Inc., 2022. URL http:://proceedingg.neei.i pager.illee/ppeer//02////// 9ee10ffc383333ccccccfffee12222 Faper-Conferencees
Ivanov, S. and Prokhorenkova, L. Boost then convolve: Gradient boosting meets graph neural networks, 2021. URL httphhttp////. http://////a...ooo//////////////
Jeffares, A., Curth, A., and van der Schaar, M. Deep learn- ing through telescoping lens: A simple model provides empirical insights grokking, gradient boosting &amp; beyond. In Globerson, A., Mackey, I.., Belgrave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 123498-123533, Curran Associates, Inc., 2024. URL /tLhtttps://prroceedd.gg
Grinsztajn, L., Oyallon, E., and Varoquaux, G. Why do
tree-based models still outperform deep learning on
paper paper.fllee/ppeer/////////l/ df33 41330022779550076677762233200303
typical tabular data? In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 507-520. Curran Associates, Inc., 2022. URL http:/sss/.pcccecnnsseeee...ese Spper_filessspaaer/22222/ 03788766221388072666778888 Haper-Datasets and Benchma rf...d..
Kadra, A., Lindauer, M., Hutter, F,, and Grabocka. J. Well-tuedd simple nets excel on tabular datasets. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 23928-23941. Curran Associates, Inc., 2021. URL paper_files/pppeerreeee paper_filles/pper//22////e co226497a9722.15544420088 Paper.pdf.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine. S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Dy, and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learn- ing. volume 80 of Proceedings of Machine Learn- ing Research, pp. 1861-1870. PMLR. 10-15 Jul 2018. URL https: w30/haarnojjalb..ee
Katzir, L.,, Elidan, G., and EI- Yaniv, R. Net-dnf: Effective deep modeling of tabular data. International Confer- ence on Learning Representations, 2021. URL https: / //opereviee.nett wwteinneeforuunni
96kho.
Hancock, J. T. and Khoshgoftaar, T. M. Survey on categori- cal data for neural networks. Journal of Big Data, 7(1): 28, Apr 2020. ISSN 2196-11 doi: 10.1 186//405377-
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q. and Liu, T.-Y. Lightgbm: A highly efficient gradi- ent boosting decision tree. Guyon, Luxburg, U. V.,

---

Gradient Boosting Reinforcement Learning
Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. /tL.ttpps//rrrcceiig...ee... paper..flees/ppper///0////// #44FfVaa022fee338666029966333333 Paper.pdf.
Lyzhin, I,, Ustimenko, A., Gulin, A., and Prokhorenkova,
L. Which tricks important for learning rank? In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato. S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learn- ing, volume 202 Proceedings of Machine Learning Research, pp. 23264-23278, PMLR, 23-29 Jul 2023. URL https: http://rrroennnnn.....e..seee.ee 1yzhin23a.htmi
Kersting, K. and Driessens, K. Non Non-parametric policy gradients: a unified treatment of propositional and re- lational domains. In Proceedings of the 25th Interna- tional Conference Machine Learning, ICML '08. pp. 456 463, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 978 9781605582054. doi: 10.1145/13901 1390214, URL hwt.h::/t/t......
Ma, H., Cao, J., Fang, Y., Zhang, W., Sheng, W., Zhang, S., and Yu, Y. Retrieval-based gradient boosting decision trees for disease risk assessment. In Pro-
ceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, pp. 3468-3476, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393880. doi: 10. 1 011.55534678.3333905 URL 10.1145/353468..5533
Kurach, K., Raichuk, A., Stanczyk, P., Zajac, M., Bachem, O., Espeholt, L., Riquelme, C., Vincent, D., Michal- ski, M., Bousquet, O., and Gelly, S. Google research football: A novel reinforcement learning environment. Proceedings of the AAAI Conference Artificial Intel- ligence, 34(04):4501-45 -51000 Apr. 2020. doi: 10.1609/ aaai.v34i04.5878 URL https:://jsss.aaaiiaiii index.php/ AAII/ attaaarricceeiieeeeeeeeee
Malinin, A.,
Prokhorenkova, L.,, and Ustimenko, Uncer- tainty in gradient boosting via ensembles, 2021. URL
https: httppss//axxvv.rrgg/ abs httppr///aab/00001555500
Mason, L., Baxter, J., Bartlett, P., and Frean, M. Boosting algorithms as gradient descent. In Solla, S., Leen, T., and Müller, K. (eds.), Advances Neural Information Processing Systems, volume 12. MIT Press, 1999. URL https::/prroceeddiggneeeee paper filles/paper/11999/E1e 9 6a33baaB90056662224499883333 Paper.pdf.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning, 2019. URL https: / //taaaaxvvvogga/aae/ http///.////500099///
London, B., Lu, L., Sandler, T., and Joachims, T. Boosted off-policy learning. In Ruiz, F., Dy. J., and van de Meent, J.-W. (eds.), Proceedings of The 26th Interna- tional Conference on Artificial Intelligence and Statis- tics, volume 206 of Proceedings of Machine Learn- ing Research, pp. 5614-5640. PMLR, 25-27 Apr 2023. URL httph: http::////oceeeeeeeeeeeee wtw.:eeddiig................eeee r.press////ee.. v206/ 1ondon23a.htm
McElfresh, D., Khandagale, S., Valverde, J., Prasad C. V., Ramakrishnan, G., Goldblum, M., and White, When do neural nets outperform boosted trees on tabular data? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 76336-76369. Curran Associates, Inc., 2023. URL attps://pprceeeeee https://prooeed.ngg..eeu..pp.p. paper filles/ppper//223///1e// [Dcdeeodfff(3300059555eeee22222 Paper-Datasets and and
Lundberg, S. M. and Lee, S.-I. A unified approach inter- preting model predictions. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30, pp. 4765-4774. Curran Asso- ciates, Inc., 2017. URL htwp::tpp/cc
Bencharrss.dddd
Meinke, A. and Hein, M. Towards neural networks that provably know when they don't know, 2020. URL
interpreting modell-oooellee weel--prriiiiinnse
https://arxiv.ort httpg//ab//19.../
Lütjens, B., Everett, M., and How, J. Certified adver- sarial robustness for deep reinforcement learning. In Kaelbling, L. P., Kragic, D., and Sugiura, K. (eds.), Proceedings of the Conference Robot Learning, vol- ume 100 of Proceedings of Machine Learning Re- search, pp. 1328 1337. PMLR, 30 Oct-01 Nov 2020. URL https: / / Jutjenn224 0a. html.
Mnih, V., Badia, A. Mirza, M., Graves, A., Lilli- crap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learn- ing. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Ma- chine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1 1937, New York, New
ml wtt.ainnssmm...ree/seesses
12

---

Gradient Boosting Reinforcement Learning
York, USA, 20-22 Jun 2016. PMLR. URL https: proceedi www.mm.rrrees/veee. ha 6.html.
Schulman, Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. In Bengio, Y. and Le- Cun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: www.i.oriiioo......a...........a abs /tt../.orrr/aaa//5602022222 /1446.02438.
NVIDIA. CUDA Toolkit Documentation, 2025. URL
https:/
toolkit. Version 12.1.
Ota, K., Jha, D. K., and Kanezaki, A. A framework for training larger networks for deep reinforcement learning. Mach. Learn.. 113(9))6113--139, June 2024. ISSN 0885- 6125. doi: 10.100774409944-222-04477 URL https: / /t//oi.org/////// 0111::000000011110-00--------000
Schulman, J., Wolski, F. Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algo- rithms, 2017. URL https: http:///axxv.ooroo
1707.06347.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z. Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., Vito, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-pprformance deep learning library, 2019. URL https: attaaaaxivviggaeeee///ee abs 1912.01703.
Seto, H., Oyama, A., Kitora, S., Toki, H., Yamamoto, R., Kotoku, J., Haga, A., Shinzawa, M., Yamakawa, M., Fukui, S., and Moriyama, T. Gradient boosting decision tree becomes more reliable than logistic regression in predicting probability for diabetes with big data. Sci- entific Reports, 12(1):15889) Oct 2022. ISSN 2045- 2322. doi: 10. 00010038/4119880022204444 URL https: wtl...org/1..00338880
2222-20149-22
Peng. X. B., Kumar, A., Zhang, G., and Levine, S. Advantage- weighted regression: Simple and scalable off-policy reinforcement learning, 2019. URL https: / //waxxi.org////eeee http:/a000111100117///
Shwartz-Ziv, R. and Armon, A. Tabular data: Deep learning is not all you need. Information Fu- sion, 81:84-90, 2022. ISSN 1566-2535
doi:
URL
Poesia, G., Dong, W., and Goodman, N. Contrastive reinforcement learning of symbolic reasoning do- mains. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 15946-15956. Curran Associates, Inc., 2021. URL https://prrccedingssneur.. paper..ilees/pperr222//// $59555c74e9afc4552 9c11e444a-6 Paper.pdf.
http:s//www.cceencccerreettcccco arriccleppi//11552552220022
Shyalika, C., Wickramarachchi, R., and Sheth, A. P. A comprehensive survey event prediction. ACM Comput. Surv., 57(3), November 2024. ISSN 0360-0300. doi: 10. 10.1145/3699955 10.1 00.1155/669995
URL https://doi.orrs
Sigrist, F. Gaussian process boosting. Journal of Machine
Learning Research, 23(232) 2022. URL http:
/ / jmlrrorr/g//////e///// wacporg//pppaeeeee http:/pppappp/////peeee
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. Catboost: unbiased boosting with cat- egorical features. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N. and Garnett, R. (eds.). Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https::/pproceddnggsneer.e paper.fflles/ppppr/////////LL 14021756b3331aacc12228828889 Paper.pdf.
Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., and Goldstein, T. Saint: Improved neural net- works for tabular data via row attention and contrastive pre-training, 2021. URL https: http:://///xvv/o//////// abos//abs/ 2106.01342.
Sutton, R. S. and Barto, A. G. Reinforcement Learn- ing: An Introduction. The MIT Press, second edi- tion, 2018. URL encomp wee.eeedeeeeee....... deas.net/ book/the teobk/thebookk--e 2nni-22dd......
Raffin, A. RI baselines3 zoo. https://gtttiii
DLR- SuR-RR/rr--baseinnest zoo, 2020.
Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. In Solla, S., Leen, T., and Müller, K. (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines:: Reliable rein- forcement learning implementations. Journal of Machine Learning Research. 22(268):1-8, 2021. URL http:
paper waper.files/paper////eee peperr/flees/pperr/99///11e
13

---

Gradient Boosting Reinforcement Learning
464d82288800000888eedc6cc5----- Paper.pdf.
Xu, K., Zhang, M., Li, Du, S. S., ichi Kawarabayashi, K., and Jegelka, S. How neural networks extrapolate: From feedforward graph neural networks, 2021. URL
Tessler, C., Givony, S., Zahavy, T., Mankowitz, D., and Man- nor, S. A deep hierarchical approach to lifelong learning in minecraft. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017. doi: 10.1609/ aaai. vaai.v31i1.10744 URL /tttpps////os...a.....rr httpss/phggp.aaaaarrraaaaeee////
https://arxiv.org http://ai.v...raaaaa/00.....1.1.
Zaběrgja, G., Kadra, A., and Grabocka, Tabular data:
Is attention all you need?, 2024. URL https:/ //
arxiv.org www.oorggbb////e http/abb...2//////
Zhao, Y., Wong, 7. S.-Y., and Tsui, K.
frame- work of rebalancing imbalanced healtheare data for rare events' classification: A case of look-alike sound-alike mix-up incident detection. Journal of Healthcare Engineering. 2018(1):6275335, 2018. doi: attpss//oonlieelllbbaaaay.eee.oo
Tian, Z., Xiao, J., Feng. H.,, and Wei. Y. Credit risk assess- ment based on gradient boosting decision tree. Procedia Computer Science, 174:150-160, 2020. ISSN 1877-0509.
URL doi: httpp://www.scceeneddieeet.co. /
URL
atticlee/sL/$317775022033384
2019
International Conference on Identification, Information and Knowledge in the Internet of Things.
abs//10.1155/211887755....
Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., Cola, G. D., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., Perez-Vicente, R., Pierré, A., Schulhoff, S., Tai, J. J., Tan, H., and Younis, O. G. Gymnasium: standard interface for reinforcement learning envi- ronments, 2024. URL https: www.i.orggg///aa..tt/..........r abs/ 2407.17032.
Ulmer, D. and Cinà, G. Know your limits: Uncer- tainty estimation with relu classifiers fails at reliable ood detection. In de Campos, C. and Maathuis, M. H. (eds.), Proceedings of the Thirty-Sevent Con- ference on Uncertainty Artificial Intelligence, vol- ume 161 of Proceedings of Machine Learning Re- search, pp. 1766-1776. PMLR, 27-30 Jul 2021.
URL
ulmer21a.htme
Ustimenko, A. and Prokhorenkova. L. Stochasticrank: Global optimization of scale-free discrete functions. In III, D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learn- ing, volume 119 of Proceedings of Machine Lear- ing Research, pp. 9669-9679, PMLR, 13-18 Jul 2020. /v19/
URL http:: wwLLhttpss/prroceeings.l./ ust:imenko20a.htt...
Ustimenko, A., Beliakov, A., and Prokhorenkova, L. Gradi- ent boosting performs gaussian process inference, 2023. URL httpth:tt///xi/eeeeesss
Wassan, S., Suhail, B., Mubeen, R. Raj. B., Agar- wal, U., Khatri, E., Gopinathan, S., and Dhiman, G. Gradient boosting for health iot federated learn- ing. Sustainabiliry, 14(24). 2022. ISSN 2071- 1050. doi: 42433944 URL https://
www.ddpi.com/20777
14

---

Gradient Boosting Reinforcement Learning
Appendix
This appendix provides supplementary materials that support the findings and methodologies discussed the main text. organized into five sections present the full experiment results, implementation details, hyperparameters used during the experiments, training progression plots, and experimental plots, respectively. These materials offer detailed insights into the research process and outcomes, facilitating deeper understanding and replication of the study.
A. Multt--Objective Training With GBT
In GBRL, we compute gradients for both the actor and critic objectives per timestep. We then concatenate these gradients: St = attto et.eritic] and use them to build a decision tree. At each candidate node split, we evaluate a score function to decide the best split. However, since the actor and critic gradients can differ in magnitude, this score can become biased toward objective. To address this, we explored two strategies:
Gradient normalization per output dimension with L2-based split scoring.
Cosine similarity, which emphasizes gradient direction over magnitude.
Both approaches are supported in our codebase and produce good results, while the cosine similarity performed slightly better (Figure 14).
500
Episodic Reward
400
300
200
0.0
0.2
0.4 Training Timesteps
0.8
1.0
x10#
Score Function
Cosine
L.2
Figure 14. Multi-objective gradient split strategics in PPO-GBRL CartPole-vI. We compare gradient normalization and cosine similarity as methods for handling actor-critic gradient imbalance when building trees. Both strategies yield good performance, with cosine similarity achieving slightly faster convergence. Curves show average episodic reward over 1M timesteps.
B. Why Do Certain Algorithms Perform Differently Than Others?
Apart from inherent algorithmic differences, the main difference for GBRL the number fitted trees per update. This analogous to the number of update steps in a neural network, since each tree represents a functional gradient step for the actor or critic. To illustrate this effect, we trained agents in CartPole-vl environment for 1M timesteps and varied the number of epochs while keeping the batch size and the rollout length fixed, thereby varying the number of updates per rollout. We repeated this experiment for several batch sizes while keeping the rollout length fixed 2048. As shown in Figure 15, increasing the number of trees added per rollout improves performance, but increases computational cost. Moreover, larger batch size also increases computational time, due to larger number of samples used to build each tree. Hence, there is a clear trade-off between computational cost and performance.
This trade-off also explains the difference in performance of various algorithms when combined with GBRL, as their update schemes inherently affect how many trees are fitted per rollout.
15

---

Gradient Boosting Reinforcement Learning
A2C- performs a single gradient step per rollout using the entire batch. In GBRL, this results in single constructed tree per rollout. Since GBTss update the value and policy functions directly, such large, coarse updates destabilize learning.
PPO in contrast, uses minibatches and multiple epochs per rollout to better control the trade-off. This allows GBRL to fit several small trees per rollout, yielding more stable and incremental updates.
AWR is an off-pooicy algorithm. Therefore, GBRL could potentially build many trees without adding sample complexity. However, we noticed that our best performing models had a huge computational cost to finish training. As result, we resorted to redueing the number of gradient steps per update at the expense performance. Future work could investigate methods for optimizing GBRL with fixed tree budget (such as pruning and distillation), which may enable the applicability of more demanding RL algorithms, such as AWR, using GBRL.
In general, as each tree approximates a gradient step, GBRL benefits from frequent updates to refine its ensemble gradually. This makes it particularly well-suited algorithms that support minibatching and multiple epochs per rollout.
C. Implementation Details and Hyperparameters
This section provides implementation details, information regarding compute resources, and tables containing the hyperpa- rameters used in our experiments enabling the reproducibility of our results. Table 2 lists GBRL hyperparameters for all experiments.
C.1. Algorithm Details
Our GBRL. method is implemented within the Stable Baselines3 framework (Raffin et al., 2021). a widely used library for standardized reinforcement learning algorithms. For comparisons, we use baseline NNs from RL Zoo3 (Raffin, 2020), which provides tuned hyperparameters per environment. For all experiments, we used standard MLP with two hidden layers, each containing 64 units and tanh activations. The hidden layers are shared between the actor and the critic, and each component has an additional separate linear layer. This architecture is the default in Stable Baselines3 for the environments we tested and is widely used as baseline in RL literature. We maintained this consistent architecture across all environments to ensure fair comparison.
A2C (Mnih et al., 2016) is a synchronous, on-policy Actor-Critie algorithm designed to improve learning stability. The critic learns a value function, V(s), used to estimate the advantage. This advantage is incorporated into the policy gradient updates, reducing variance and leading smoother learning. The policy updated using the following gradient: (TO(TT) [V, & log (e(a
(a As)
AWR (Peng et al., 2019) is an off-policy Actor-Critic algorithm. Provided dataset D, AWR updates both the policy and the value through supervised learning. This dataset can be pre-defined and fixed (offline), or continually updated using the agent's 's experience (replay buffer). At each training iteration k; AWR solves the following two regression problems:
V44 arg
arg min E. a.. [I| a) (s))
V
T&+1 arg max E |log (a s) expl 1 3 (s Au a))l
where G(s, a) represents the monte-carlo estimate or TD(A) of the expected return (Sutton & Barto, 2018).
Advantage computation: We Generalized Advantage Estimation (GAE) proposed by Schulman et al. (2016) to compute the advantage: A, Ei=0 (1))) where 6, rz + ($++))) - (s+)). The GAE parameters and for each environment are provided in Table 2 of the appendix. Critic: We use on-policy actor-critic algorithms, where the critic predicts the value function (not action-value Q). We use a standard L2 loss between the critic's prediction and the target return estimate: Lenitie (0) = ||Va (s:) - Gella, where G. is either a Monte Carlo return estimate or a bootstrapped n-step return, depending on the specific algorithm (PPO, A2C, or AWR).
C.2. Environments
Classic-Contas Control & Box2D. We evaluate GBRL against NNs on all available classic-control tasks provided in Gymnasium (Towers al., 2024) and on v2 This includes three discrete action space environments: Acrobot-vl, CartPole-
16

---

Gradient Boosting Reinforcement Learning
Episodic Reward
500
10e
Batch Size: 64
Runtime [min]]
400
300
101
200
100
10P
0.00
0,25
0.50
0.75
1.00
Training Timesteps
500
Batch Size: 512
Episodic Reward
Runtime [min]
104
300
200
104
100
10
0.00
0.25
0.50
0.75
1.00
Training Timesteps
x10
500
Batch Size: 1024
Episodic Reward
10a
Runtime [min]
300
100
10a
0.00
0.25
0.50
0.75
Training Timesteps
x10P
500
10a
Batch Size: 2018
Episodic Reward
400
300
Runtime
10a
100
10a
0.25
0.75
Training Timesteps
x10
Updates per rollout
16
64
256
32
128
Figure 15. Effect of number of gradient updates per rollout on PPO-GBRL's performance in CartPole-vl. We fix the batch size and the number samples collected per rollout, while varying the number PPO epochs, thereby changing the number gradient updates per rollout. This improves sample efficiency, with more epochs leading to faster convergence. Results are shown over 1M timesteps. The number samples collected per rollout was fixed to 2048 for all experiments.
v1, MountainCar-vO, and two continuous action space environments: Pendulum-vi v and CoumtainCarrCmmiiuuuuu We trained on both environments for 1M steps (1.5M for LunarLander-v2), For NNs we used the hyperparameters provided in RL. Baselines3 Zoo (Raffin, 2020), except for AWR where we followed the hyperparameters described in the paper.
17

---

Gradient Boosting Reinforcement Learning
Google Research Football & Atari ram. The Football domain (Kurach et al., 2020) consists of vectorized 115- dimensional observation space that summarizes the main aspects of the game and 19 discrete actions. We focus on its academy scenarios, which present situational tasks involving scoring single goal. A standard reward of +1 is granted for scoring, and we employed the "Checkpoints" shaped reward structure. This structure provides additional points as the agent moves closer towards the goal, with maximum reward of 2 per scenario. The Atari-ram domain consists of a vectorized 128-dimensional observational space representing the 128-byte RAM state and up to 18 discrete actions. We trained agents in both domains for 10M timesteps.
MiniGrid. The MiniGrid environment (Chevalier-Boisver et al., 2023) is a 2D grid world with goal-oriented tasks requiring object interaction. The observation space consists of 7x7 image representing the grid, a mission string, and the agent's direction. Each tile in the observed image contains 3D tuple dictating an object's s color. type, and state. All MiniGrid tasks emit a reward of +1 for successful completion and 0 otherwise. We trained our NN-based agents flattened observation space using the built-in one-hot wrapper, specified by RL Baselines3 Zoo. For GBRL agents, we generated a 51-dimensional categorical observational space by encoding each unique tile tuple as a categorical string to represent the observed image. Categorical features were added for the agent's 's direction (up, left, right, down) and missions. All agents were trained for 1M timesteps, except for PutNear, FourRooms, and Fetch tasks, which were trained for 10M based on the reported values for PPO NN in RL Baselines3 Zoo.
Variable Isolation Environment. The goal of the variable isolation task is to separate a target variable within linear equation. In the basic task variation, the initial state is linear equation of the form ar + b = c, where a, b, c are digits within the range [1 9] that can be either positive or negative. The action space is multi-discrete, where the agent chooses an action type. the digit to use, and the sign of the digit. The action types are: addition, subtraction, multiplication, and division. The agent receives reward proportional to the number of steps taken in the episode if it successfully isolates otherwise the reward is O.
In the two wwriabiiblle linear equation variation, the initial state is a linear equation of the form az + by + c = d. and the goal is to isolate y/ on the left side. The actions are equivalent to the basic task, with the addition or subtraction of a digit multiplied by x. Finally, in the balanced two-variable linear equation variation, the initial state a linear equation of the form az b- = ey d. and the goal is to isolate y on the left side. The actions are equivalent to the two-variable linear equation variation with the addition of the operations involving y. In all the environment variations, the agent observes the coefficients as inputs. For the two variable variations, we add a small reward bonus if the agent successfully moves a variable or a constant to the correct side. We trained GBRL for 15M steps and NNs for 30M steps.
C.3. Compute Resources
All experiments were done on the NVIDIA NGC platform on a single NVIDIA V100-32GB GPU per experiment. Training time and compute requirements vary between algorithms and according to hyperparameters. The number of boosting iterations has the largest impact on both runtime and memory. GBRL experimental runs required from 1GB to 24GB of GPU memory. Moreover, runtime varied from 20 minutes for 1M timesteps training on classic environments and up to days for 10M timesteps on Atari-ram. NN experimental runs required up to 3GB of GPU memory and runtime ranged from 10 minutes and up 3 days. The total compute time for all experiments combined was approximately 1800 GPU hours. Additionally, the research project involved preliminary experiments and hyperparameter tuning, which required an estimated additional 168 GPU hours.
D. Detailed Result Tables
This section contains tables presenting the mean and standard deviation of the average episode reward for the final 100 episodes within each experiment. More specifically, Table presents results for Continuous Control & Block2D environments, Tables 4 and 5 present results for the high-dimensional vectorized environments, and Table 6 presents results for the categorical environments.
E. Learning Curves
This section presents learning curves depicting model performance throughout the training phase. Figures 16 to 19 show the training reward as a function of environment steps of the agents trained in the experiments. The column order is: A2C,
18

---

Gradient Boosting Reinforcement Learning
batch
clip range
cnef
gae lambda
gamma
num epochs
steps
policy
value Ir
Acrobot
512
0.2
0.0
0.94
0.99
20
128
0.16
0.034
CariPole
64
0.2
0.0
0.8
0.98
1
128
0.029
0.015
Lnder ander
256
0.2
0.0033
0.98
0.999
20
512
16
0.031
0.003
MountainCar
256
0.2
0.033
0.98
0,999
20
512
16
0,031
0.003
MountainCar Continuous
256
0.2
0.033
0.98
0.999
20
512
16
0.031
0.003
Pendulum
512
0.2
0.0
0.93
0.91
20
256
0.031
0.013
Football
512
0.2
0.0
0.95
0,998
10
256
0.033
0.006
Atari-Ram
0.92
8c-5
0.95
0.99
512
0.05
0.002
MiniGrid
0.0
0.95
0.99
20
0.17
0.01
Variable Isolation
0.62
0.08
0.95
0.99
20
256
0.18
0.006
(a) PPO. For continuous action spaces, used log std init -2 and log std Ir lin 0.0001.. We utilized gradient clipping for Gym
environments. Specifically, 10 for the value gradients and 150 the policy gradients.
ent coef
gae lambda
gamma
num steps
num envs
policy Ir
value Ir
log std init
log std Ir
Acrobot
0.0
1
0.99
8
4
0.79
0.031
CartPole
0.0
0.99
16
0.13
0.047
Lunarl Lundrl.ander
0.0
1
0.995
32
0.16
0.04
MountainCar
0.0
I
0.99
16
0.64
0.032
MountainCar Continuous
0.0
1
0.995
128
16
0.0008
2.8e-6
0
0.0004
Pendulum
0.0
0.9
0.9
10
32
0.003
0.056
-2
0.00018
Football
0.0004
0,95
0.998
8
0.017
Atari- Rii-RRam
0.0009
0,95
0.993
128
8
0.17
0.013
MiniGrid
0.0
0.95
0.99
128
0.34
0.039
(b) A2C
batch coef'
gae lambda
gamma
train froq
gradient steps
policy
value
log init
std
Acrobot
1024
0.0
0.95
2000
150
0.1
CartPule
1024
0.0
0.95
0.99
2000
150
0.05
0.1
Lunarl ander
1024
0.0
0.95
0.99
2000
150
0.05
0.1
MountainCar
64
0.0
0.95
0.99
2000
150
0.64
0.032
MountoinCar Continuous
64
0.0
0.95
2000
150
0.083
lio.0.0017
Pendulum
1024
0.9
0.07
0.0005
Football
512
0.03
0.95
0.99
750
10
0.09
0.00048
Atari Ram
1024
0.0
0.95
0.993
2000
50
0.0779
0.0048
MiniGrid
1024
0.0
0.95
0.99
1500
25/100*
0.0075
0.005
(c) AWR. For all envs, buffer size 50,000, 8 0.05. *MiniGrid environments used 100 gradient steps for tasks trained for 1M steps, and 25 gradient steps tasks trained for 10M steps, reduced tree size.
Table 2. GBRL hyperparameters NN represented by an MLP with two hidden layers.
Table Continuous-Cone Control and Box2D environments: Reported values are the mean and standard deviation of the average episade reward over the final 100 training episodes, aggregated across 5 random seeds.
Acrobot
CartPole
LunarLander
MountainCar
MountainCar Continuous
Pendulum-vi
NN: A2C
82,27 + 3.29
500.00 i 0.0
43.01 +: 106.26
148.90 1: 24.10
92.66 1: 0.32
188.64 +: 22.32
GBRL: A2C
-90.73 + 2.98
500.00 0.0
47.93 + 41.00
-124.42 5.74
93.15 + 1.19
-538.83 + 66.25
NN: AWR
102.53
57.25
500.00 0.0
+ 282.48
160.65
53.97
18.93 42.34
+ -159.64
GBRL: AWR
-118.12
33.54
497.54 + 3.11
76.03 56.62
144..88 + 24.53
44.38 + 45.94
-1257 98.10
NN: PPO
74.83 1 1.22
500.00 1 0.0
261.73 1 6.93
115.53 1.39
85,81 1. 7.51
249.31 1 60.00
GBRL: PPO
-87.82
+ 2.16
500.00 0.0
248.72 + 59.10
- -110.55 + 15.60
89.42 + 5.73
-246.89 = 20.61
AWR, and PPO.
19

---

Gradient Boosting Reinforcement Learning
Table Football Academy environments: Reported values are the mean standard deviation average episode reward over the final 100 training episodes, aggregated across 5 random seeds.
3 vs with keeper
Corner
Counterattack Easy
Counterattack Hard
Empty Goal
Empty Goal Close
NN: A2C
1.78
1.00 0.17
1.58 +: 0.35
1.43 + 0.17
1.93 + 0.05
0.0
GBRL: A2C
1.59 + 0.17
1.01 0.07
1.11 + 0.14
1.00 + 0.05
1.81 + 0.03
2.00 0.00
AWR
1.50 0.37
0.04
0.36
0.21
1.90 +:
1.92 0.17
GBRL: AWR
1.66
0.05
0.95
0.92 = 0.05
1.93 0.07
2.0
NN: PPO
1.61 + 0.05
0.95 0.02
1.43 + 0.15
1.23 + 0.18
1.98 + 0.01
1.99 0.00
GBRL:
1.63 + 0.19
L.05 0.20
1.64 +
1.23 = 0.07
1.84 + 0.06
2.0
Pass Shoot keeper
Run Pass
Shoot keeper
Run
Run score w/ keeper
Single Goal Lazy
NN: A2C
1.41 + 0.37
1.77 0.08
1.87 0.12
1.25 0.23
1.65 + 0.04
GBRL: A2C
1.60 0.21
1.60 0.14
1.82 0.10
1.15 0.08
1.31 0.11
NN: AWR
1.26 + 0.46
1.15 + 0.14
1.81 + 0.14
1.25 + 0.34
1.28 + 0.27
GBRL: AWR
1.35 4: 0.37
1.53 +: 0.40
1.98 + 0.01
0.99 +: 0.16
1.03 +: 0.12
NN: PPO
1.31 0.13
1.64
1.91 0.09
1.13 0.06
1.68 0.09
GBRL: PPO
1.87 + 0.09
1.85 0.08
1.83 1.83 0.04
1.95 0.02
1.73 0.06
Table AtaiirammNFrammssipp environments: Reported values are the mean and standard deviation of the average episode reward the final 100 training episodes, aggregated across random seeds.
Alien
Amidar
Asteroids
Breakout
Gopher
NN: A2C
1802.24 + 323.12
304.62 + 55.61
2770.46 + 271.97
76.69 + 30.08
3533.84 = 118.50
GBRL: A2C
595.08 43.51
48.71 14.65
1402.66
161.67
11.52
502.20 341.88
NN: AWR
739.82
303.06
86.32 + 40.16
2308.68 + 257.72
26.57 1 9.91
1471.93 + 716.65
GBRL: AWR
829.99
166.48
125.53 1: 25.25
1592.63
109.96
17.32
913.06 79.95
NN: PPO
1555.32 + 107.59
310.93 + 80.13
2309.46 + 145.66
32.88 + 15.74
2507.84 + 108.37
GBRI: PPO
1163.86 76.54
186.32
50.63
1514.34 317.46
19.96
1215.04 81.01
Kangaroo
Krull
MsPacman
Pong
SpaceInvaders
NN: A2C
2137.6 + 425.64
9325.38 777.12
2007.64 + 116.52
15.39 +'2 4.26
462.30 35.56
GBRL: A2C
948.8 + 483.80
5291.4 433.35
989.68 + 100.02
-12.80 11.10
265.36 44.61
NN: AWR
1214.8 + 313.42
4519.78 522.11
892.31 + 289.36
-10.25
2.11
842.00
130.51
GBRL: AWR
1809.26 37.51
6419.26 + 387.76
1641.84 + 284.19
11.68 + 3.79
397.85 + 566.38
NN: PPO
2487.4 829.65
9167.3 294.30
2069.22 22 202.48
18.50
1.60
479.77 65.07
GBRL: PPO
2160.8 + 826.92
6888.66 + 756.18
2069.22 + 538.62
15.40 + 6.55
434.84 31.83
20

---

Gradient Boosting Reinforcement Learning
Table MiniGrid environments: Reported values are the mean standard deviation of average episode reward over the final training episodes, aggregated across 5 random seeds.
DoorKey-5x5
Empty-Randon-5x5
Fetch-5x5-N2
FourRooms
GoToDoor-5x5
NN: A2C
0.96 0.00
0.77 = 0.42
0.43 = 0.03
0.62 0.19
0.05 + 0.04
GBRL: A2C
0.96 0.00
0.96 0.00
0.62 + 0.02
0.51 + 0.07
0.78 + 0.02
NN: AWR
0.57
0.52
0.96 0.00
0.90 + 0.26
0.19 + 0.12
0.95 + 0.01
GBRL: AWR
0.96 + 0.00
0.97 = 0.00
0.95 + 0.01
0.54 + 0.05
0.91 + 0.01
NN: PPO
0.78 0.40
0.96
0.00
0.89 + 0.03
0.53 + 0.03
0.60 + 0.06
GBRL: PPO
0.96 0.00
0.96 = 0.00
0.96 + 0.01
0.56 + 0.04
0.96 + 0.00
KeyCorridorS3R1
PutNear-6x6-N2
RedBlueDoors-6x6
Unlock
NN: A2C
0.75 + 0.42
0.01 + 0.00
0.30 + 0.22
0.77 + 0.43
GBRL: A2C
0.39 0.48
0.18 = 0.018
+
0.90 0.09
NN: AWR
0.93 + 0.00
0.60 + 0.13
0.83 + 0.00
0.96 0.00
GBRL: AWR
0.94 0.00
0.36 0.01
0.84 + 0.03
0.95 0.00
NN: PPO
0.76 +
0.001 0.00
0.17 + 0.40
0.97 0.00
GBRL: PPO
0.95 0.00
0.44 0.19
0.88 + 0.02
0.97 0.00
21

---

Gradient Boosting Reinforcement Learning
100
1000
100
Acoilnt
2000
200
-200
-000
400
400
400
2000
0011
0.13
0.2
0.4
0.8
0.88
1.0
0.11
0.2
D-4
1.6
0.5
1.0
14.02
0.2
0.4
3.8
0.5
1.9
X300
X10
600
5000
CurPole4]
300
200
200
X.16
X
1
3000
Lanol Laneeandderv
-100
100
0.75
x
x
X10
120
1210
100
MousiainCurn
-1.00
1:40
160
-200
160
-200
-200
0.11
01:2
0.4
0.5
0.M
1.0
0.10
0.2
0.4
1.6
0.8
1.0
03.0
0.2
0.3
3.8
0.6
3.0
Mountiin CemumtinrCCCCCiiiiii. CoConnnnossuu
50
00
50
-50
0.5
0.6
0.8
1.0
x.109
X/16
- 200
Pendolamevi
400
100
-00
730
9070
500
1000
-1200
1350
1200
Timesqpi
x100
Timestent
x100
Timesteps
GBRL.
AZCGBRL
PPOGIBII
Figure 16. Classic Control and Box2D environments: Training reward function of environment steps. Reported values the and standard deviation of the average episode reward over the final 100 training episndes, aggregated across 5 random seeds.
22

---

Gradient Boosting Reinforcement Learning
Comar
Chante Branke-Attick
Counter-Attel
Hord
Einpty
Closs
Empry
Pass and Sboot with
Rui, Pars and Shoet
Scone
Rue Sccre with
with vs
0.0
0.4
Timastaps
Timestaps
x200
x107
GBRL
A2CNN
Figure 17. Football Academy environments: Training reward function of environment step. Reported values the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across 5 random seeds.
23

---

Gradient Boosting Reinforcement Learning
1250
2000
1000
1600
Alen
Amidar
200
200
50
Asconais
1000
1800
1000
1000
100
00.
Breakout
.40
6000
3000
1000
2000
1000
1000
1000
300
2000
1000
1000
10000
6000
7500
Krail
6000
4000
MrPacman
2000
1500
2000
1000
1000
1000
201
Ponz
10
-15
10
-20
Spacolracers
1000
400
760
300
250
0.2
Timestaps
Timestaps
x107
Timestaps
x110
AWR
GBO
A2COBRL
A2CNN
Figure Atari-ramNo www.rraooooFaaiiivvevr environments: Training reward a function of environment step. Reported values the mean and standard deviation of the average episode reward over the final 100 training episndes, aggregated across 5 random seeds.
24

---

Gradient Boosting Reinforcement Learning
DoodKey Sx5
0.4
0.6
1.0
0.0
0.2
0.0
0.5
1.0
0.0
0.0
0.4
0.8
1.0
Timestops
1119
Timesteps
x10
Timesteps
x10
Empgy Random 3x5
0.36
0.5
0.5
0.01
00.0
0.33
02.4
10.0
0.8
1.0
0.01
0.2
0.44
0.0
0.6
1.0
0.01
0.2
0.4
15.6
0.5
2.3
Timesteps
10
Timestops
x
Timesteps
Fx5c N2
1.0
0.9
0.4
0.5
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.0
0.8
1.0
Timestops
Timesleps
X300
Timesteps
X110
Fourkcoms
0.50
0.25
0.26
0.00
0.00
1.0
0.0
Timmssopss
x00
Timesteps
x100
Timmestops
x10
GoTnDoor 5x5
1.0
1.0
0.3
0.5
0.3
Timesteps
X100
Timesteps
X
Timesteps
x110
KeyConions:8g
1.0
0.5
0.0
0.2
Timestops
x10
Timesteps
x
Timessipps
PutNoar6o6
0.2
0.5
0.50
0.26
0.9
0.00
Timestops
Timesteps
Timestops
RedBlneDoors 5x6
1.0
0.5
0.0
0.0
0.00
0.2
D-4
D.6
0.K
1.0
0.11
0.2
13.4
0.6
0.8
1.01
0.11
0.2
0.44
0..6
1..8
Timestops
Timesteps
x
Timesteps
X.10R
Unoock
0.5
0.5
0.3
0.3
0.0
0.0
0.2
0.4
0.C
0.9
1.0
0.0
0.2
0.4
0.0
0.5
1.0
0.0
0.2
0.4
0.0
0.8
1.0
Timestaps
X11
Timesteps
X10
Timesteps
XT11
AWR GBRL
A2C GBRL
PPONN
AWR NN
A2CNN
PPO GBRL
Figure 19. MiniGrid environments: Training reward as a function of environment step. Reported values are the mean and standard deviation of the average episode reward over the final 100 training episodes, aggregated across random seeds.
25

---

Gradient Boosting Reinforcement Learning
-
PutNear- -x-NN---
PutNear-6x6- 6x6-N22v0
1.0
Allocated GPU Memory [%)
Frames per Second
Reward
40
100
0.5
20
0.0
0.0
0.5
1.0
0.0
0.5
1.0
200000
400000
Timesteps
x107
Timesteps
x107
Boosting Iterations
Fetch-5x5-N2-v0
Fetch-5x5-N2-vv0
Fetch-5x5-N2-v0
1.0
GPU Memory [%]
Frames per Second
Allocated
50
100
Reward
0.5
25
50
0.0
0.0
0.5
1.0
0.0
0.5
1.0
200000
400000
Timesteps
x107
Timesteps
x107
Boosting Iterations
FourRooms-v0
FourRooms-v0
FourRooms-vo
1.0
GPU Memory [%)]
Frames per Second
Allocated
100
Reward
0.5
25
50
0.0
0.0
0.5
1.0
0.0
0.5
1.0
200000
400000
Timesteps
x107
Timesteps
x107
Boosting Iterations
Shared AC
Separate AC
Figure 20. Sharing actor critic tree structure significantly increases efficiency while retraining similar performance. Training reward, GPU memory usage, and FPS, compared across 10M environment (5 seeds, 3 MiniGrid environments)
26


Reviews:
[{"review_id": "tWV05lEAvF", "reviewer": "Reviewer_is4y", "summary": "The authors introduce a reinforcement learning method based on gradient-boosted trees (Gradient Boosting Reinforcement Learning, GBRL). \n\nThe method consists of gradually training an additive ensemble of decision trees to output both a policy and Q-value estimates at the leaves. The loss function has two terms: a cumulative reward term for supervising the actor (i.e. policy outputs of the trees) and an L2 term for supervising the critic (which the authors don’t specify). \n\nThe gradient of the cumulative reward with respect to the policy outputs can be computed using the policy gradient theorem, with the Q-value predictions being used to estimate the advantage function. The critic uses an L2 loss, whose gradient is the difference between the critic’s prediction and the target. Hence, the functional gradient can be computed from the tree’s outputs, enabling the application of the usual gradient-boosted trees approach (Section 4.1).\n\nThe authors then experimentally validate GBRL, comparing it to neural network methods on on:\n\n- Standard RL environments: GBRL exhibits equivalent or superior performance on grid environments, a football environment, and continuous control environments. It is inferior in the Atari RAM domain, which exposes raw system memory states in Atari games. The authors attribute this underperformance to the fact that decision trees rely on single-feature splits, while NNs can compose raw features into more informative representations.\n\n- A custom environment for assessing robustness to signal dilution: the environment consists of simple algebraic manipulations on equations whose complexity can be tuned. The rewards in this environment are sparse. GBRL converges significantly faster than NN based methods, and achieves higher reward.\n\n- A custom environment for assessing spurious correlations: the environment consists of a grid world where the agent is meant to pick up a red ball. The authors experiment with placing a red box at various locations near the target ball to distract the agent. They find that GBRL exhibits greater out-of-distribtion performance when training on one box location and testing on other box locations. \n\n- Variations of the aforementioned environments to assess robustness to state perturbations: the authors consider (a) introducing an additional confounder object in the grid world mentioned above, to assess robustness to irrelevant information, (b) introducing Gaussian noise in classical control tasks to assess robustness to corruption/noise, and (c) varying the number of players in the football environment mentioned above, to assess robustness to missing features. In all cases, they find GBRL generalizes better than the NN baseline.\n\nThe authors provide a CUDA-based implementation of their method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. Below we address your points, which we will also clarify in the revised paper.  \n \n**Stable Baselines3 and RL Zoo3**: Our GBRL method was implemented within the Stable Baselines3 framework, a widely-used reinforcement learning library providing standard implementations for RL algorithms [1]. We compared GBRL against baseline neural networks (NNs) provided by RL Zoo3 [2], which include tuned hyperparameters for each environment, ensuring a fair and standardized evaluation.  \n**NN architecture** : For all experiments, we used a standard MLP with two hidden layers, each containing 64 units and tanh activations. This architecture is the default in Stable Baselines3 for the environments we tested and is widely used as a baseline in RL literature. We maintained this consistent architecture across all environments to ensure fair comparison.\n\n- **Advantage computation**: We use Generalized Advantage Estimation (GAE) as proposed by Schulman et al. (2018) [3] to compute the advantage: $A_t = \\delta_t + (\\gamma\\lambda)\\delta_{t+1} + \\ldots + (\\gamma\\lambda)^{T-t-1}\\delta_{T-1}$, where $\\delta_t = r_t + \\gamma V(s_{t+1}) - V(s_t)$. The GAE parameters $\\gamma$ and $\\lambda$ for each environment are provided in Table 2 of the appendix.  \n- **Critic**: We use on-policy actor-critic algorithms, where the critic predicts the value function (not action-value Q). We use a standard L2 loss between the critic's prediction and the target return estimate: $L_\\text{critic}(θ) = ||V_\\theta(s_t) - G_t||^2$,  where $G_t$ is either a Monte Carlo return estimate or a bootstrapped n-step return, depending on the specific algorithm (PPO, A2C, or AWR). \n\nBoth the GAE and target return calculations were done using built-in functions in Stable_baselines3. \n\n**Multi-objective training with GBT**: In GBRL, we compute gradients for both the actor and critic objectives per timestep. We then concatenate these gradients: $g_t := [g_{t, \\text{actor}}, g_{t, \\text{critic}}]$ and use them to build a decision tree. At each candidate node split, we evaluate a score function to decide the best split. However, since the actor and critic gradients can differ in magnitude, this score can become biased toward one objective. To address this, we explored two strategies:\n- Gradient normalization per output dimension with L2-based split scoring.\n- Cosine similarity, which emphasizes gradient direction over magnitude.  \n \nBoth approaches are supported in our codebase and produce good results, while the cosine similarity seems to perform slightly better.   The results are visualized for the Cartpole environment in the following link: https://pasteboard.co/qXY0zekTq7uM.png.  We will add this result as part of an ablation study to the appendix of the final version.\n\n\n\n[1] Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. \"Stable-Baselines3: Reliable Reinforcement Learning Implementations.\" Journal of Machine Learning Research 22 (2021): 1–8.\n\n[2] Raffin, A. \"RL Baselines3 Zoo.\" GitHub repository, 2020. https://github.com/DLR-RM/rl-baselines3-zoo\n\n[3] Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. \"High-Dimensional Continuous Control Using Generalized Advantage Estimation.\" arXiv preprint arXiv:1506.02438 (2015).", "claims_and_evidence": "I found the major claims of the paper (especially w.r.t. GBRL’s robustness) to be well-justified by the experiments.", "methods_and_evaluation_criteria": "Yes, the proposed method makes sense for online RL.", "theoretical_claims": "N/A (there are no theorems, propositions, lemmas or proofs in the paper).", "experimental_designs_or_analyses": "Experiments are extensive, covering both standard RL environments and tasks designed specifically to assess various types of robustness. This paints a much clearer picture of GBRL’s strengths than if the authors had only evaluated it in standard benchmarks.", "relation_to_broader_scientific_literature": "Prior work of Brukhim et al. (2022) (A Boosting Approach to Reinforcement Learning) had considered the application of boosting to RL, but from a theoretical perspective. In addition, Kersting & Driessens (2008) introduced Non-Parametric Policy Gradients (NNPG), which serve as a starting point for GBRL. GBRL also builds on foundational RL methodologies by using an Actor-Critic training setup.", "essential_references_not_discussed": "I am not aware of any glaring omissions of references to prior work.", "other_strengths_and_weaknesses": "Strengths:\n\n- Clear presentation of key motivations and results in the experiments section.\n\n- Extensive experiments, covering both standard RL environments and tasks designed specifically to assess various types of robustness. This paints a much clearer picture of GBRL’s strengths than if the authors had only evaluated it in standard benchmarks.\n\n- Provision of a CUDA-based library for GBRL (at least as far as the authors claimed), which should enable practitioners and researchers to more easily test GBRL in their domains of interest.\n\n- More generally, the experiments and running time considerations (appendix A.3) indicate GBRL is competitive with baseline RL methods in terms of performance and training cost, while showcasing certain robustness methods. This expands the toolbox of practitioners when doing RL in e.g. noisy or nonstationary environments.\n\nWeaknesses:\n- It was not immediately clear from reading the paper what architectures are used for the baseline neural networks in the experiments section. It would help contextualize the results if the authors could be clearer about this in the main paper.\n\n- Lack of clarity in the methods section regarding (a) how the advantage in Eq. 3 is computed and (b) how exactly are the Q-value prediction heads supervised.\n\n- Assuming e.g. a temporal difference loss for the critic, I would imagine the final loss used for obtaining the tree would be different from the one presented in Eq. (4). More generally, it would be good to contextualize how multi-objective training works when dealing with trees, and any additional constraints relative to working with neural networks.", "other_comments_or_suggestions": "While there is already an inherent amount of non-stationarity in the training data when doing online RL (as changes in the policy produce changes in the training data), it would have been good to see an experiment directly comparing GBRL’s behavior under non-stationarity relative to baseline NNs.", "questions_for_authors": "1. What NNs are used as a baseline in each experiment?\n1. Would it be possible to give more detail (e.g. via pseudocode) on (a) how exactly the advantage is computed in the policy gradient and (b) what loss is used to train the critic?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pE66sSi0Uu", "reviewer": "Reviewer_KuLP", "summary": "The paper proposes GBRL, a framework that adapts gradient boosting trees (GBTs) to reinforcement learning (RL) tasks. Recognizing that neural networks (NNs) often struggle with structured and categorical features, the authors leverage the natural strengths of GBTs—namely, their ability to handle heterogeneous data and capture abrupt transitions in decision boundaries. Furthermore, to adapt the GBTs to the dynamic nature of reinforcement learning, the paper designed the GBRL framework to continuously interleaving tree construction with environment interaction. The paper presents extensive experiments on a range of tasks, demonstrating that GBRL can outperform or match NN-based approaches in settings where the input features are naturally structured, while also showing increased robustness to out-of-distribution states, signal dilution, and spurious correlations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment of our paper, recognizing the comprehensive experiments and the value of our approach for structured data in RL. We address each of your concerns below and will revise the paper accordingly.  \n\n**Regarding the \"unbounded growth of the ensemble as the policy improves\"**:  \nYou raise a valid concern about the scalability of our approach. In our current implementation, ensemble size does increase with training steps.\nThere are several possible approaches to address this limitation:\n- **Tree Pruning**: One can reduce memory requirements through pruning redundant or low-importance splits after training. \n- **Ensemble Compression**: One can distill knowledge from large ensembles into smaller, more efficient tree structures.\n- **Adaptive Growing Strategy**: Rather than adding trees at fixed intervals, one can add them selectively based on performance plateaus or gradient magnitudes.  \n\nWhile we briefly mentioned these approaches in the paper, we will expand the limitations section to further explain these approaches as future research directions.\n\n**Regarding \"additional ablations on hyperparameter sensitivity\"**:   \nThis is an excellent suggestion. We performed these experiments, provide them below, and will add them to the paper. Specifically, we propose to add an ablation study of GBRL-based PPO in the appendix examining: \n- **Learning rate ablation for policy and value function components**: Here, we keep one fixed while changing the other. Similar to training with neural network estimators, we observe that large learning rates harm stability, whereas low learning rates result in very slow convergence. The results can be seen in the following links: https://pasteboard.co/PxZtndOMFuzZ.png, https://pasteboard.co/2W5UT93RBNrn.png.\n\n- **Tree depth limitations**. This ablation shows that tree depth has a large impact on convergence, however deeper trees lead to increased runtime and compute cost. A larger tree depth corresponds to a better gradient approximation, which may explain the faster convergence shown in these results. In the paper, we chose a maximal tree depth of 4, a value that balances performance (the ability to solve environments) with maintaining a reasonable wallclock time-to-converge. The results can be found in the following link: https://imgur.com/iu0IUsT.\n\n- **Batch size**. Using a rollout length of 2048 samples, we analyze the impact of PPO batch size on performance. We observe that batch size significantly impacts convergence. Specifically, smaller batches result in GBRL building more trees per rollout, improving adaptability. However, smaller batches also lead to noisier gradient estimates due to limited samples per constructed tree (blue, 16 samples per batch, starts much faster, but fails to converge to a stable solution). Conversely, larger batches stabilize training by reducing variance through averaging within leaves (more samples per utilized per constructed tree), but build fewer trees per rollout (gray, 2048 samples per batch, is very stable but also very slow to converge). Hence, both excessively small and large batch sizes negatively impact performance. Results can be found in the following link: https://pasteboard.co/ENlUg1ffghZU.png.\n\n\n**Regarding the lack of \"in-depth theoretical analysis\"**:  \nPlease see our detailed response to **Reviewer 6bS7** regarding the lack of theoretical proof of convergence and handling non-stationarity.\n\n[1] Mei, Jincheng, et al. \"On the global convergence rates of softmax policy gradient methods.\" International conference on machine learning. PMLR, 2020.  \n[2] Cortes, Corinna, Mehryar Mohri, and Dmitry Storcheus. \"Regularized gradient boosting.\" Advances in neural information processing systems 32 (2019).  \n [3] Beygelzimer, Alina, et al. \"Online gradient boosting.\" Advances in neural information processing systems 28 (2015).  \n [4] Hu, Hanzhang, et al. \"Gradient boosting on stochastic data streams.\" Artificial Intelligence and Statistics. PMLR, 2017.  \n[5] Kersting, Kristian, and Kurt Driessens. \"Non-parametric policy gradients: A unified treatment of propositional and relational domains.\" Proceedings of the 25th international conference on Machine learning. 2008.", "claims_and_evidence": "yes.", "methods_and_evaluation_criteria": "yes.", "theoretical_claims": "yes (no proofs).", "experimental_designs_or_analyses": "yes, the experiments are sound.", "relation_to_broader_scientific_literature": "The paper encourage to exploration and usage of gradient booting tree algorithms in the domain of deep reinforcement learning.", "essential_references_not_discussed": "no.", "other_strengths_and_weaknesses": "**Strength**:\n* The proposed GBRL framework was shown to outperform traditional PPO methods implemented in \"stable baseline\" in many tasks.\n* Comprehensive experiments are conducted to support the claims of the authors, providing many valuable insights.\n* The paper are well-written and sound.\n\n\"Weakness\":\n* A significant limitation is the unbounded growth of the ensemble as the policy improves. While the paper discusses potential solutions (e.g., tree pruning or ensemble compression) for future work, this issue might affect long-term or real-time deployment and is not resolved in this paper.\n* Although the experiments are comprehensive, additional ablations on hyperparameter sensitivity and design choices (such as the impact of learning rates for policy vs. value functions) would strengthen the empirical claims.\n* Though comprehensive experiments are conducted, the papers lacks in-depth theoretical analysis to further support the strength of GBT concretely.", "other_comments_or_suggestions": ".", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No."}}, {"review_id": "hWwJLAIB2Z", "reviewer": "Reviewer_sYtv", "summary": "This paper presents a GPU accelerated method to use gradient-boosting trees for use in RL. The contribution is in the fact that existing methods for boosting-trees are tailored to offline learning, whereas, the authors' method is fully incremental. They then show empirically many benefits of using tree based function approximation in RL settings such as, robustness to spurious correlations and input perturbations. The algorithm is RL-loss agnostic, can use 'weight sharing' (or more accurately '(sub-)tree sharing'), and compares well against a neural network baseline.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive feedback. We address your main points below and will incorporate the suggested revisions in the final version.  \n\n**Regarding line 71-72 and performance on structured tasks**:\nWe agree with this point. We will revise the contribution statement (lines 71-72) to: \"In popular environments, our framework demonstrates competitive performance against NNs while showing particular strengths on structured tasks.\"  \n\n**Regarding the critique on NNs (lines 350-353)**:\nWe agree that any model class—including NNs and GBTs—can be made to fail or succeed depending on the setup. Our intent is to highlight a key difference in inductive biases. As discussed in Jeffares et al. (2024) [1], the kernel representations induced by GBTs are bounded and often behave more predictably on irregular or out-of-distribution inputs, while NN-tangent kernels can be unbounded and vary significantly on test points far from the training distribution. While techniques like LayerNorm can help stabilize neural network activations, they don't fundamentally change the difference in inductive biases and they often don't fully solve the extrapolation issues in OOD scenarios. We will revise the relevant paragraph in the paper to reflect this more clearly and avoid overgeneralization. \n\n**Why do certain algorithms perform differently than others?**   Apart from inherent algorithmic differences, the main difference for GBRL is the number of fitted trees per update.  \n- **A2C** - performs a single gradient step per rollout using the entire batch. In GBRL, this results in a single constructed tree per rollout. Since GBTs update the value and policy functions directly, such large, coarse updates can destabilize learning.  \n- **PPO** -  in contrast, uses minibatches and multiple epochs per rollout. This allows GBRL to fit several small trees per rollout, yielding more stable and incremental updates.  \n- **AWR** - is an off-policy algorithm. Therefore, GBRL could potentially build many trees without adding sample complexity. However, we noticed that our best performing models had a huge computational cost to finish training. As a result, we resorted to reducing the number of gradient steps per update at the expense of performance. Future work could investigate methods for optimizing GBRL with a fixed tree budget (such as pruning and distillation), which may enable the applicability of more demanding RL algorithms, such as AWR, using GBRL.   \n\nIn general, as each tree approximates a gradient step, GBRL benefits from frequent updates to refine its ensemble gradually. This makes it particularly well-suited to algorithms that support minibatching and multiple epochs per rollout. We will include a targeted PPO-based ablation study in the appendix to demonstrate the impact of minibatching performance. Results can be found in the following link: https://pasteboard.co/ENlUg1ffghZU.png.  \n\n**Other improvements**:  \nWe greatly appreciate your detailed comments on presentation and formatting issues. We will implement all of your suggested improvements to enhance the clarity, readability, and technical accuracy of our paper.  \n\n**Regarding specific questions**:\n1. **Figure 5; which RL algorithm was used for GBRL and NN, PPO? What neural network architecture was used?**\nWe compared GBRL to NNs using PPO and a standard MLP with two hidden layers, each containing 64 units and tanh activations. This architecture is the default in Stable Baselines3, and a common choice in RL literature, for these environments. As such, we used the same NN architecture for the Equation environment. We used PPO for all experiments in section 5.2. \n\n2. **Figure 11; how come AWR-GBRL performs so poorly on pendulum-v1?**\nAWR-GBRL performs poorly on Pendulum-v1, despite working well on many other environments. Since PPO-GBRL and AWR-NN succeed in this task, we believe the issue lies in the optimization process specific to AWR. Although we did try to tune hyperparameters for this setup, it's likely that the configuration remains suboptimal. \n\n\n[1] Jeffares, A., Curth, A., and van der Schaar, M. \"Deep Learning Through a Telescoping Lens: A Simple Model Provides Empirical Insights on Grokking, Gradient Boosting & Beyond.\" Advances in Neural Information Processing Systems, 37, 2024.", "claims_and_evidence": "The authors describe a variety of benefits for using tree-based function approximation, all of which they address in their experiments. Limitations for their method are also shortly discussed. \n\nSo the claims match well with what is shown. However, I do think that e.g., line 71-72 column 1 deserves a more nuanced discussion. The results are simply not decisive enough to conclude superior performance on structured tasks.", "methods_and_evaluation_criteria": "No complaints.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental setup is exhaustive, explained well, and aligns with the claims that the authors want to make.\nHowever, much of the discussion of the results can be presented more scientifically rather than as the current salespitch. Especially in section 5.1. \n\n**For example:** Line 240 column 2, the authors praise results from the appendix on PutNear, Four Rooms, and Fetch. Figure 14, however shows that this is not consistent over RL algorithms and note that only 5 seeds are shown. The GBRL method fails on RedBlueDoors, KeyCorridor with A2C, and is slower to learn with AWR compared to the NN. But it does seem to often outperform the NN when paired with PPO. \n\nWhy not summarize much of the current discussion and shed some light on why some RL losses yield strong differences in performance? This would improve impact so practitioners understand how GBRL interacts with such choices.\nAlthough, I doubt that this question can be answered with the currently low number of seeds...", "relation_to_broader_scientific_literature": "I am not very familiar with prior work that uses gradient boosted trees in RL. \n\nThe authors already cite the work by Abel D., et al. (2016) and Brukhim N. et al. (2023).\n\nHowever, there have also been approaches for decision tree learning within RL settings, see Silva et al. (2020) or Vos D. (2023). These are not mentioned in the paper right now, but could strengthen the 'tabular data' paragraph in the related work for concurrent tree-based approaches within RL. \n\n---\n\nSilva, A., Gombolay, M., Killian, T., Jimenez, I. &amp; Son, S.. (2020). Optimization Methods for Interpretable Differentiable Decision Trees Applied to Reinforcement Learning. <i>Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics</i>. PMLR. 108:1855-1865 \n\nVos, D., & Verwer, S. (2023). _Optimal decision tree policies for Markov decision processes_. In _Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence_ (pp. 5457–5465). IJCAI. [https://doi.org/10.24963/ijcai.2023/606](https://doi.org/10.24963/ijcai.2023/606)", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "## Strength\n\nLanguage, wording, structure, style, and coherence is excellent. This is a well written paper that should be understandable for a broad audience.\n\nLimitations of the method and implementation are clearly formulated.\n\nIn general, there is not too much to critique about this paper. There are indeed still limitations, however the aim, motivation, and approach are all very clear. The authors also provide a codebase that is implemented in CUDA and compatible with commonplace baseline repositories. This paper should be accepted as I think it establishes a valuable starting point for future work.\n\n---\n\n## Weaknesses\n\nThe large GPU memory requirements and inefficient compression of old trees does not make this algorithm super competitive. Although, it does have redeeming features that future work should directly explore.\n\nLast sentence of the conclusion is too salesmen-like, and can be more nuanced.\nGBRL is a nice alternative to existing solutions. However, I don't see it as a 'step toward' real-world tasks. It rather feels like a sidestep.\n\nThe critique from 350-353 on NNs is a bit arbitrary, you can always come up with examples that break particular models. If I use LayerNorm within my NN, then the output will always be in a reasonable range even if OOD. This can be formulated better.", "other_comments_or_suggestions": "Line 85 column 2, use \\citet instead of \\citep for 'Kersting & Driessens (2008) ...'.\n\nSection 3, why are the variables bold-face? To indicate a vector? This confused me for a moment since the reward $r$ is also bold-face, which made me think we were dealing with a multiobjective problem. Also, in figure 1 the state $s_t$ is not bold while in the main text it is. So, I suggest a slight revision for consistency and clarity.\n\nRemove 'superior' from line 312 column 2.\n\nFontsize of plot ticks and labels in figures 5 are too small. The same goes for the ticks in figures 3, 8 and 9.\n\nThe title/ y-labels in figure 6 is a bit confusing, I suggest to right-align the current y-labels and modify the ysuplabel as 'episodic reward'.\n\nLine 272-273 column 1 is difficult to read.\n\nLine 380-384 column 2 broken sentence.\n\nSummary in line 425-428 column 1 belongs in the conclusion (obsolete).\n\nAt times there are singular words being split up from their text due to the format, this obstructs reading. Please revise the following,\n - Line 290 column 2, 'dilution'\n - Line 365 column 2, '$s_t = s_t + \\epsilon |s_t|$'\n - Line 416 column 1, 'missing.'\n\nMany references are formatted inconsistently, wrong, or do not have proper journals or venues listed. Examples: \n - Line 482 column 1, URL uses a DOI link.\n - Line 458 column 2, Duan T. (2020) is a ICML 2020 paper.\n - Line 471 column 2, Gorishniy Y. (2023) is a NeurIPS 2021 paper.", "questions_for_authors": "Figure 5; which RL algorithm was used for GBRL and NN, PPO? What neural network architecture was used?\n\nFigure 11; how come AWR-GBRL performs so poorly on pendulum-v1 ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1QxywU3t2K", "reviewer": "Reviewer_6bS7", "summary": "This paper presents **Gradient Boosting Reinforcement Learning (GBRL)**, an attempt to integrate **Gradient Boosting Trees (GBT)** into reinforcement learning (RL). GBTs are known for their performance in structured data tasks, but RL has long relied on deep neural networks. The authors propose modifications that allow GBTs to function effectively in RL.\n\nThe main idea is to adjust standard GBT updates so they can handle the **dynamic and non-stationary nature** of RL. Instead of training separate networks for policy and value functions, GBRL uses a **single decision tree ensemble** for both, which reduces memory consumption. \n\nExperiments show that GBRL **performs well in structured environments like MiniGrid**, is **competitive in continuous control tasks**, but **struggles with high-dimensional unstructured data** such as Atari-RAM. The authors also implement **GPU acceleration**, making large-scale training feasible.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and valuable feedback. Below we address each of your concerns.  \n\n1. **Lack of theoretical proof of convergence & handling non-stationarity**: While comprehensive theoretical analysis is beyond the current scope, GBRL builds upon established foundations from both gradient boosting and policy gradient theory.\nEach tree in our approach minimizes a functional gradient step, making GBRL's convergence analysis potentially related to known policy gradient convergence results. The recent work on softmax policy gradient convergence rates [1] shows O(1/t) convergence for tabular settings with true gradients. We believe similar analysis could potentially apply to GBRL, as our tree-based approximation maintains the key properties that enable convergence -- specifically, GBRL preserves the policy gradient's directional information while providing a piecewise approximation of the advantage landscape.\nFurthermore, GBT methods have convergence guarantees in various settings. Recent theoretical works have analyzed convergence for GBT on convex losses [2], and online gradient boosting methods have known convergence guarantees under certain conditions [3,4]. The work on non-parametric policy gradients [5] also provides a foundation for our approach.\nA full theoretical analysis would require examining how the tree-based approximation interacts with policy improvement dynamics and whether it preserves the Łojasiewicz condition identified in [1]. We believe our empirical evidence of convergence across diverse environments provides strong practical support for GBRL while setting the stage for future theoretical analysis.  \n\n2.  **Poor performance on high-dimensional unstructured tasks (Atari-RAM)**: We agree with your observation. GBRL's weaker performance in high-dimensional raw-feature spaces (such as Atari-RAM) arises from a fundamental difference in inductive biases between decision trees and NNs [6, 7]. Decision trees partition the input space according to single feature thresholds, whereas NNs learn complex feature compositions via continuous transformations. This makes NN more suited for environments where useful information is embedded in combinations of features. This highlights the complementary nature of GBRL to existing NN-based methods rather than positioning it as a replacement. We will expand our discussion in the revised paper to explicitly highlight this limitation and its implications.  \n\n3. **Lack of comparisons to specialized architectures (TabNet, SAINT)**:  The goal of this work is to combine the popular GBT tool with RL, analyze its performance, and overcome the challenges that this combination introduces. One major advantage that we found is GBT's robustness to OOD scenarios compared to NNs, and not only handling structured data. While specialized architectures like TabNet/SAINT excel at purely tabular data, our environments span a broader range of settings including those with mixed features, temporal dependencies, and various action spaces. Our primary focus was therefore on comparing with the standard MLP-based approaches most commonly used in RL rather than models optimized solely for tabular data. This allows us to assess GBRL's performance in the broader context of RL applications.  \n\n4. **Computational cost**:  You raise a valid concern about the scalability of our approach. Please see our detailed response to **Reviewer KuLP** regarding the unbounded growth of the ensemble as the policy improves.\n\n\n[1] Mei, Jincheng, et al. \"On the global convergence rates of softmax policy gradient methods.\" International conference on machine learning. PMLR, 2020.  \n[2] Cortes, Corinna, Mehryar Mohri, and Dmitry Storcheus. \"Regularized gradient boosting.\" Advances in neural information processing systems 32 (2019).  \n [3] Beygelzimer, Alina, et al. \"Online gradient boosting.\" Advances in neural information processing systems 28 (2015).  \n [4] Hu, Hanzhang, et al. \"Gradient boosting on stochastic data streams.\" Artificial Intelligence and Statistics. PMLR, 2017.  \n[5] Kersting, Kristian, and Kurt Driessens. \"Non-parametric policy gradients: A unified treatment of propositional and relational domains.\" Proceedings of the 25th international conference on Machine learning. 2008.  \n[6] Grinsztajn, Léo, Edouard Oyallon, and Gaël Varoquaux. \"Why do tree-based models still outperform deep learning on typical tabular data?.\" Advances in neural information processing systems 35 (2022): 507-520.  \n[7] Rahaman, Nasim, et al. \"On the spectral bias of neural networks.\" International conference on machine learning. PMLR, 2019.", "claims_and_evidence": "The claims regarding GBRL’s superior performance in structured RL tasks are well-supported by experiments. However, the claim that GBRL can be a general alternative to deep RL models is not fully justified, as results on high-dimensional tasks like Atari-RAM show significant performance gaps.\n\n## update after rebuttal  \nThanks for the response. I will keep my score.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are generally appropriate. The selection of structured and continuous control tasks aligns well with the goal of testing GBTs in RL. However, including more comparisons against structured-data-oriented models could improve the fairness of the evaluation.", "theoretical_claims": "The paper lacks formal theoretical guarantees. There is no proof of convergence, and the theoretical discussion on stability in non-stationary RL environments is minimal.", "experimental_designs_or_analyses": "The experimental setup is well-structured and diverse, covering different RL environments. However, failure cases, particularly in high-dimensional settings, are not deeply analyzed.", "relation_to_broader_scientific_literature": "N/A.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Strengths\nOne clear strength is **novelty**—very few works have tried scaling gradient boosting for RL. The **shared tree-based actor-critic model** is an interesting way to cut down memory requirements. The paper’s **experimental coverage** is also broad, evaluating performance across structured, continuous control, and high-dimensional tasks.\n\nGBRL also **enhances interpretability in RL**. Unlike deep networks, decision trees provide a transparent way to understand decision-making, which could be crucial for real-world deployment.\n\nAdditionally, **practicality** is a strong point. GBRL integrates with existing RL frameworks like **Stable Baselines3**, lowering the barrier for adoption.\n\nWeaknesses\nHowever, there are some issues. **Theoretical justification is lacking**—there’s no proof of convergence, and the paper does not fully explore how GBTs handle non-stationary RL data. While GBRL does well in structured environments, **its performance in unstructured tasks is poor**, especially on **Atari-RAM**, and the authors don’t offer a deep explanation of these failures.\n\nAnother limitation is **baseline comparisons**. The experiments mostly compare GBRL to standard MLP-based RL models but **ignore specialized architectures** like TabNet or SAINT that are designed for structured data.\n\nLastly, **computational cost is high**. GBRL requires **significantly more GPU memory** than MLPs (~24GB vs ~3GB in some cases), making it less practical for resource-constrained environments.", "other_comments_or_suggestions": "The paper would benefit from a more detailed discussion on why GBRL struggles in high-dimensional settings and an analysis of computational trade-offs in real-world deployment.", "questions_for_authors": "Please refer to the weaknesses section for key concerns regarding theoretical justification, baseline comparisons, and failure case analysis.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper is well motivated with powering RL with another alternative, boosting trees, instead of NN. The authors provided good feedbacks to the reviews. Although it’s uniformly agreed between reviewers is a solid contribution. The empirical results of the paper is a little weak. First, the experiment domains are too simple. Second, the performance on other benchmarks like Atari need still be improved. Please also try to address the points made by the reviewers.
Decision: Accept (poster)