Is Value Functions Estimation with Classification Plug-and- play for Offline Reinforcement Learning?
Authors: Denis Tarasov
Kirill Brilliantov
Dmitrii Kharlapenko
Venue: Accepted by TMLR ()

Abstract: In deep Reinforcement Learning (RL), value functions are typically approximated using deep neural networks and trained via mean squared error regression objectives to fit the true value functions. Recent research has proposed an alternative approach, utilizing the cross-entropy classification objective, which has demonstrated improved performance and scalability of RL algorithms. However, existing study have not extensively benchmarked the effects of this replacement across various domains, as the primary objective was to demonstrate the efficacy of the concept across a broad spectrum of tasks, without delving into in-depth analysis. Our work seeks to empirically investigate the impact of such a replacement in an offline RL setup and analyze the effects of different aspects on performance. Through large-scale experiments conducted across a diverse range of tasks using different algorithms, we aim to gain deeper insights into the implications of this approach. Our results reveal that incorporating this change can lead to superior performance over state-of-the-art solutions for some algorithms in certain tasks, while maintaining comparable performance levels in other tasks, however for other algorithms this modification might lead to the dramatic performance drop. This findings are crucial for further application of classification approach in research and practical tasks.

Published in Transactions on Machine Learning Research (11/2024)
Is Value Functions Estimation with Classification Plug-and- play for Offline Reinforcement Learning?
Denis Tarasov
tamasod@ethh.cc
Deportment of Computer Science
ETH Zürich
Kirill Brilliantov
kbrilHianto@dddha.ckk
Deportment of Computer Science
ETH Zürich
Dmitrii Kharlapenko
dkharlapenko@ethh.cct
Department of Computer Science
ETH Zürich
Reviewed on OpenReview: https: // openreview. net/ net/forum?
forum? ? id= MFJ
Abstract
In deep Reinforcement Learning (RL), value functions are typically approxinated using deep neural networks and trained via mean squared error regression objectives to fit the true value functions. Recent research has proposed an alternative approach, utilizing the cross-entropy classification objective, which has demonstrated improved performance and scalability of RL algorithus.. However, existing study have not extensively benchmarked the effects of this replacement across various domains, the primary objective was to demonstrate the efficacy of the concept across broad spectrum of tasks, without delving into in-depth analysis. Our work seeks to empirically investigate the impact of such a replacement in offiine RL setup and analyze the effects of different aspects performance. Through large-scale experiments conducted across a diverse range of tasks using different algorithms, we aim to gain deeper insights into the implications this approach. Our results reveal that incorporating this change can lead to superior performance over state-of-the--rt solutions for some algorithms in certain tasks, while maintaining comparable performance levels in other tasks, however for other algorithms this modification might. lead to the dramatic performance drop. This findings are crucial for further application of classification approach in research and practical tasks
1 Introduction
In the realm of deep Reinforcement Learning (RL), the conventional approach approximating value functions has long relied employing the Bellman optimality operator alongside mean squared error (MSE)
regression objectives, owing to the continuous nature of the task at hand. However, insights from other
domains of machine learning have illuminated the potential benefits of employing classification objectives
even in scenarios where regression seems as a natural choice (Rothe et al. 2018 Rogez et al.
2019 This shift
has been attributed to various hypotheses, including the stability of gradients [mani et al.
2024 improved
feature representation (Zhang et al. 2023), and implicit biases (Stewart al. 2023)
QQur code available https://githhb. com/DT6A/C1ORL

---

Published in Transactions on Machine Learning Research (11/2024)
While the use of regression loss has yielded remarkable results in value-based RL (iiverr et al. 2017 it
also presents certain challenges and limitations (Kumar et al. 2020a; 2021 Agarwal et al. 2021 Lyle et al.
2022j. Notably, recent research Farebrother et al. (2024) has demonstrated that replacing regression with classification for training value functions offers several advantages, including enhanced scalability, feature
representation, performance, and robustness to noisy largets and non-stationarity. While this study broadly explores the effects of this replacement, it does not delve deeply into the case of implementation or the impact of newly introduced hyperparameters.
Offiine RL (Levine al. 2020 represents rapidly developing RL subfield, wherein the objective is to train an agent using pro-collected dataset without direct interaction with the environment. In recent. years, numerous algorithms have been developed to address this setup (Kunar et al. 2020b Fujimoto & Gu 2021 Kostrikov et al. 2021] An et al., 2021 Chen et al., 2021 Akimov et al. 2022 Yang et al.) 2022
Ghasemipour et al., 2022 Nikulin et al. 2023 Tarasov al., 2024a). The majority of these algorithms fall
under the category of off-policy value- based approaches, owing their alignment with the problem sctup's
requirements.
Given the prevalence of off-policy value-based approaches in offline RL, it becomes imperative to delve deeper into the potential impact of employing classification for value function training within this domain. While Farebrother et al. (2024) offer some experimental insights into offline RT. tasks, we contend that there remain significant gaps to be addressed. Therefore, our study seeks to contribute to this area by conducting a thorough investigation into the utilization of classification objectives for value function training in offline RL scenarios.
This study aims to address the following questions through large-scale experiments conducted large range of tasks from the standard D4RL benchmark (Fu et al. 2020):
Is classification a 'plug-and-play replacement' for offline RL algorithms, and how does it impact performance?
Does the use of classification objectives facilitate a more robust hyperparameter search?
What is the impact of the hyperparameters introduced with classification?
Does classification enable more efficient scaling of dense neural networks compared regression?
2
Preliminaries
2.1
Offline Reinforcement Learning
The RL problem is conventionally framed Markov Decision Process characterized by a tuple (8, (8, P, RR,),, where: S C IR" denotes the state space, A C R" represents the action space, P S x A + S is the transition function, R: 5 x A + R is the reward function, and 7 E (0, 1) is the discount factor. The objective of RL is to find a policy π : 5 + that maximizes the expected sum of discounted rewards: E 7 R(s,, ac). This entails the policy learning process, where the agent interacts with its environment by observing environmental states, taking actions in response, and receiving corresponding rewards.
Offline RL presents departure from the traditional RL setup in that the agent relies solely pre-collected datasel D collected by external agents. This paradigm introduces novel challenges, such as the estimation of value functions for out-of citooof-dstrtttiinn state-action pairs, thus giving rise to an entire subfield dedicated to addressing these challenges.
2.2 Q-function objective and classification
The Q-function stands as a pivotal concept in valuo-based RL, representing the expected return of a policy starting from state 8 and taking action ae: Q(*,, a.) = Ex E> x R(5+4k ((4x))s), ar). Typically, value functions are trained using the Bellman optimality operator:
(fQ) a. R(s, a) + + max Q(s ,',
2

---

Published in Transactions on Machine Learning Research (11/2024)
Here, s' denotes the state observed after executing action a at state 8, and 0 (also known target network) represents a copy with delayed updates of the parameter 0 which parametrize the value function. During each training step, parameters / are adjusted using temporal difference (TD) error with mean squared error (MSE):
TDMSE(() = Ep ((T))()) ., a,, 0))
We adopt the framework proposed by Farebrother et al. (0024) for replacing MSE with cross-entropy loss. The approach involves parameterizing the Q-function as a distribution over returns, segmented into m bins with widths Ç, where the first bin corresponds to the predefined Uman value, and the last to Umar- The scalar value of the Q-function then computed as:
m
a(s,a,)) Σ a a, 0)o,,
where Pi represents the probability of the i-th bin, and ò; denotes the corresponding bin value. This formulation allows us to express the TD error using cross-entropy and utilize it for updating the value function parameters:
m
TDC ED Σ po (s, a, 0) log as
Farebrother et al. (2024) demonstrated that the HL-Gauss method (Imani & White 2018 is particularly effective for mapping continuous values into bins within this framework. HL-Gauss employs normal distribution analog for mapping values into neighboring bins, with a hyperparameter a determining the distribution's breadth. The authors recommend tuning the ratio a/C as a more interpretable hyperparameter, with a default value of 0.75 chosen based on statistical considerations and it has shown good empirical result.
3 Methodology
Our methodology is straight forward: we select several offline RL algorithms and adapt them for cross-entropy loss, as outlined Section [2.2] By default, we determine the values of Umin and umar by computing all possible discounted returns within a given dataset and identifying the minimum and maximum values, which aligns naturally with the offline RL setup. Herein, we provide brief descriptions of the algorithms employed in our study.
We selected three algorithms from the most prominent families of offline RL approaches: policy regularization, implicit regularization, and Q-function regularization. These categories encompass the majority of widely- recognized offline RI. methods, as noted in prior work (Levine et al. 2020 making them representative choices for illustrating the general applicability of our framework. By selecting these specific algorithms, we aim to cover diverse approaches offline RL while demonstrating that our classification loss can seamlessly integrate with various methods across these categories. The choice of particular algorithms is based on results from Tarasov et al. (2024b) where wide range of offline RL algorithms were compared against cach other. We provide pseudocodes for the algorithins with cross-entropy loss in Appendix B
Revisited BRAC (ReBRAC). ReBRAC [arasovv et al. 2024a) stands as a minimalist, state-of-the ensemble-free algorithm for both offline and offline-to-onling RL. employs MSE to penalize deviations from actions present in the dataset. Built upon TD3+BC (Fujimoto & Gu 2021), ReBRAC incorporates several modifications that significantly enhance performance. The Q-loss function takes the form:
E (s.0,e' d' (Q(s,a, 0) [[[(s, a) + (Q(( - 2)))))²
where 3 denotes penalty weight. Although only the target network part differs, the bin mapping is conducted in the same manner, with the penalty subtracted beforehand. We select ReBRAC as an exemplary algorithm with policy regularization (i.c. forcing the policy to commit actions simillar to datasot) due its high performance, simplicity, and its encounter with the Q-function divergence problem while solving certain D4RL Maze tasks.
3

---

Published in Transactions on Machine Learning Research (11/2024)
Implicit Q-Learning (IQL). IQL (Kostrikov al. 2021 represents another competitive offinee and oflline-to-online RI, amm. Its key advantage lies in its exclusion of out-of-distribution examples during training, a departure from most other offiine RL approaches. This is achieved through the training of the V-fumction: V(s)) = E 7 R(SL-K, a(+))ss)]. The Q-function loss takes the form:
+
Once again, the target component allows for classification without additional manipulations. Regularization is achieved through specially designed V-function loss, without explicit penalties for the policy or value functions. We opt for IQL as it is the best example of algorithms within this family.
Large-Batch SAC (LB-SAC). LB-SAC (Nikulin et al. 2022 is an instance of the ensemble-based SAC-N algorithm (An et al. 2021 utilizing large-batch optimization to reduce ensemble size. SAC-N comprises N Q-functions, each with the following loss:
[R(s, a) + min Q(s' 0,)))
j=
No additional adjustments are necessary to replace MSE with cross-entropy. The rationale behind this objective lies in the assumption that if Q-values follow a normal distribution ((,, then:
N ( 8 (N=1 +1
min ((s.a,0))~~((s)) ~ - J=L.N
a(ssa) -
basedd approaches are known for their efficacy example of offline RL algorithm with Q-function regularization. We opt for LB-SAC instead of the original SAC-N due to computational constraints that arisc due to the huge ensemble sizes (up to 500 critics) required in original SAC-N.
many offline RL tasks. According to the
above statement, SAC-N be also considered
4 Experimental Results
4.1 Experimental setup
We conducted experiments using three sets of tasks from the D4RL benchmark Fuu et al. Gym-MuJoCo, AntMaze, and Adroit. We utilized all datasets within each set.
2020
we employed the best hyperparameters as outlined in Tarasov et al. 2024a When exploring parameter search in isubsection 4.3] we utilized the same hyperparameter grids for ReBRAC and IQL. as in Tarasov et al. (2024a) and used a custom grid for LB-SAC due to computational canstraints. For comprehensive overview of the experimental details and hyperparameters, refer Appendix and Appendix C.
For ReBRAC in subsection 4.2
The evaluation protocol is also taken from Tarasov et. al. 20244)), where hyperparameters search is done using four random seeds and separate set of seeds is used for the evaluation with the exception that we utilized ten random seeds for the final evaluation for ReBRAC and IQL and only four seeds for LB-SAC, due to computational constraints. Note, that because of the chosen evaluation protocol tuned hyperparameters might perform worse than non-tuned which characterizes the sensitivity the random initialization.
4.2 Is classification plug-and-play.
Our initial goal was to substitute MSE with cross-cntropy without modifying any other aspects of the algorithms. We fixed the number of bins m to 101, representing a reasonable number of classes. We set a/C = 0.75 by default as was proposed by Farebrother et al. 2024)). Results for this modification are presented in the CE columns of Table for Gym-MuJoCo, Table 2 for AntMaze, and Table 3 for Adroit. We also provide rliable (Agarwal al. 2021) metrics which support all of the further claims in more readable way in Appendix I| Tt. is worth noting that, for ReBRAC in the AntMaze tasks here and further, we used
4

---

Published in Transactions on Machine Learning Research (11/2024)
large-batch optimization, similar to that used in Gym-MuJoCo, which was previously hindered by Q-function divergence.
The original ReBRAC algorithı exhibited minimal performance variation with the introduction of cross- entropy in Gym-MuJoCo tasks, except for the random dataset where a notable performance drop was observed. However, on average, the score remained relatively stable. Tn the case of AntMaze, where ReBRAC faced Q-function divergence issues, the introduction of classification mitigated this problem, resulting in notable improvement in average performance. However, in Adroit, where ReBRAC struggled with overfitting due to small dataset sizes, classification did not alleviate this issue and led to decreased performance across most scenarios.
In contrast, both IQL and LB-SAC experienced a significant performance drop in Gym-MuJoCo tasks. IQL's performance in AntMaze dramatically decreased, rendering the algorithm unable to solve medium and large tasks with this modification. Mcanwhile, LB-SAC's performance did not change significantly for AntMaze but improved for the umaze task. Notably, in the Adroit task, IQL's performance was significantly boosted in the pen environment, with relatively minor changes observed in other scenarios. However, LB-SAC's performance dropped across most tasks.
Our primary hypothesis for explaining the successful application of classification with ReBRAC and its failure with IQT. and LB-SAC is that ReBRAC heavily relies on policy regularization. Consequently, plugging classification into Q networks does not strongly affect its offline RL component compared to the impact observed in IQL or LB-SAC.
Table 1: Average normalized score over the final evaluation and ten (four for LB-SAC) unseen random seeds on Gym-MuJoCo tasks. "he" stands for "halfchootah', 'hp" stands for 'hopper', "wl" stands for "walkerdd', 'r' stands for 'random', "m' stands for 'medium', "c' stands for "expert", 'me' stands for 'medium-expert", 'mr' stands for 'mediuu-replay', "fr' stands for 'full-replay" + denote stds across random seeds. MSE denotes original algorithm implementation, CE denotes the replacement of MSE with cross-entropy, CE+AT denotes cross- cotssyyntropy with tuned algorithm parameters, CE+CT denotes cross-entropy with tuned classification parameters ReBRAC original scores are taken from Tarasov et al. (024a)).
ReBRAC
IQL
LB-SAC
Took
MSE
CE
CK-AT
CE4CT
MSE
CE
CE+AT
CE4CT
MSE
CE
CE+AT
CE4CT
1.5
13.4
0.8
13.0
0.8
18,9
1.0
28.2 +
10.0
11.6
11.1
he-m
05.6
1.0
0.7
63.6
1.4
582
11.2
19.5
1.1
42.5
: 0.3
12.6 0.4
438 1 0.3
61. 4 1.3
56.7
: 2.1
63.0
0.7
55.5 + 2.7
hose
105.9
+ 1.7
1032 6.6
101.2
100.7
4.2
96.8
2.2
92.9
+ 0.2
92.9 + 0.4
R5.5 + 0.2
105.0 +
1.5
103. + 1.0
106.0
+ 1.3
104.5
+ 2.2
he
101.1
5.2
103.5
4.
103.
5.5
10L.
6.4
92.3
2.4
86.5
4.0
13.3133
89.0 1.8
104.5
1 24
105.4
1 2.0
106.2
0.0
107.2
51.0
53.1
30.9
52.
55.4
54.4
hc ft
82.1 1211111 1.1
83.2 1. 1.5
82.4 1
1.9
83.4
0.9
75.5
0.5
62.6 J. 1.1
64.5 4 1.5
74.1 4. 0.6
79.0
1 2.0
50.7
11 0.3
SL. 1 1.0
$2.
hp
2.4
$.1.1.0. 0.9
8.9
1: 2.1
9.1
1.4
6.0
2.8
14.2 1 2.8
13.7 1 3.3
13.8 1. 7.5
14.5 1 11.5
8.2 1 2.2
9.6
4 0.4
10,9 4 233
hp
102.0
1.0
9.4
10L7
1.6
98.1
1.5
54.9
4.4
$4.1
2.0
52,9
55.5 1 1.8
90.0 1 27.5
1
10.5 1 6.2
7.8
1.0
hp-e
100.1
8.3
117.9
4.9
107.9
4.9
110.8
109.4
110.5
110.5
110.2 4 1.0
21.6 + 19.7
65.9
+ 61.9
12.4
18.1
hp me
107.0
1. 6.4
111.6
1. 0.5
111.2
1 0.3
111.6
0.5
$8.5
16.5
64.0
1 10.2
69.3 1. 23.9
102.0 1. 8.6
111.3
1. 0.3
14.9 J. 7.0
12.5
1: 7.2
34.7
37.8
lip-ur
98.1
+ 5.3
18.77
3.0
9A.1 + 12.5
96.7
3.0
96.9
64
7L.
10.1
27.2 4.5
07.2 4: 8.8
63.0 + 48.1
08.9 + 42.8
100.6 = 0.5
87.4
29.5
hp-St
107.1
108.2
108.3
107.2
63.6
50.2
4.5
4.4
12.1 911
7:22
A: 2:1
9.3 A 7.6
21.7
20.1 =:
21.8 =
21.6
wh-m
82.5
+: 3.6
85.1 2.7
85.1 2.7
86.7
1.0
83.2 = 1.2
79.9 +: 1:5
81.5 +
$0.9 + 1.5
80.3 + 6.3
89.6 + 10.7
98.7 + 7.5
98.0 + 1.5
wl-
112.3 #: 0.2
112.7 0.1
112.7 1.1
132.7
111
118.8
0.2
108.5 0.2
108.5 +: 0.2
109.5 +: 0.2
114.2 + 0.4
19.8 +: 40.0
107.6 = 1.2
112.5 = 0.2
111.7
111.9
91.0
wlone
77.3 7.9
45.2
6.7
$4.1 6.9
84.0
5.0
82.0
7.9
69.1
8.8
07.2
5.1
82.6
92.6
+ 2.7
90.9 + 6.3
96.3 +
99.8
1.9
wi-f
102.2 4: 1.7
101.8
+ 5.6
106.7
5.0
102.4
2.3
97.7
1.4
86.8 4: 3.5
83.9 + 1.9
93.6 +: 1.0
102.1 + 1.0
110.4 = 1.9
100.4 = 1.8
110.5 + 2.4
81.2
80.6
81.5
80.6
74.1
61.6
62.0
76.9
74.9
57.8
69.5
69.6
4.3 Algorithms hyperparameters search with classification
The subsequent step in our investigation involved determining whether better performance could be achieved by tuning the hyperparameters of the original algorithms. The evaluation of the best hyperparameter sets can be found under the CE+AT columns in Table 1 Table 2 and Table 3 (see also Appendix I)
In Gym-Mu.de tasks, these hyperparameter adjustments yielded slight improvements in ReBRAC's average performance, although random datasets remained problematic. Similarly, in Adroit tasks, performance saw
These notations are reused further.
5

---

Published in Transactions on Machine Learning Research (11/2024)
Table 2: Average normalized score over the final evaluation and ten (four for LB-SAC) unseen random seeds on AntMaze tasks. 'um' stands for 'umaze", 'med' stands for 'medium', "Irg" stands for 'large', 'p" stands for 'play". 'd' stands for 'diverse'. denote stds across random seeds. ReBRAC original scores are taken from Tarasov et al. )024a)
ReBRAC
IQL
Task
CR+ATT
CE++CT
MSB
CR
CR++AT
CR-CT
MSE
CE+AT
CE+CT
87.8 + 1.0
98.3
+: 2.1
98.0 # 1.6
98.8 +
79.1
2: 6.6
30.0 2: 4.6
49.6
4: 8.1
53.0 d: 6.2
18.25 4: 36.8
41.0 4: 33,2
38.3 = 13.4
57.5 1:
12.7
88.3
13.0
93.0
94.5
72.5
4.5
47.4
0.0
+ 0.5
0.0
med-p
84.0 4
88.0
6.3
00.5 3.8
88.
+: 5.1
73.8
022 0.4
0.1
1 0.3
0.4 0.6
0.0
4 0.0
0.0 0.0
0.0
med d
76.3 4 13.5
84.8
4 9.3
70.6
14.8
90.0
1: 0.8
70.8
4 3.8
0.4 1 0.9
0.3
1 0.6
0.3
1:3.:00 0.4
0.0 0.0
0.0 4: 0.0
0.0 0:0:1:0 0.0
0.0 4 0.0
hg-p
2611
86.9
6.3
7.0
0.0
0.0
0.0 0.0
Irg-d
64.4
+ 25.1
87.1
4.1
86.6
4.5
81.9
8.2
23.7
5.6
0.0
4 0.0
0.0
0.0
0.0
+ 0.0
0.0 0.0
0.0 +
Avg
76.8
89.4
38.3
90.1
00.8
16.5
16.2
17.0
2.0
6.8
6.0
9.5
Table 3: Average normalized score over the final evaluation and ten (four for LB-SAC) unseen random seeds Adroit tasks. 'ham" stands for "hammer', "rel" stands for 'relocate', "h" stands for "human', 'c" stands for 'cloned', "e" stands for "expert". + denote stds across random seeds. ReBRAC original scores are taken from
Tarasov et al. 2024a
ReBRAC
IQL
LB-SAC
Took
MSE
CE
CE-AT
CE4CT
MSE
CR
CB+AT
CE+CT
MSE
CE
CE+AT
CE+CT
pen h
100.5 = 14.1
1023 +: 10.2
85,8 # 13.5
97.5 = 13.3
21.1 16.8
108-2 = 3.4
107.2 = 33.0
4.5 2.2.2
1.1 5.6
20.1 # 19.6
5,8
pense
91. = 21.7
90.3 + 14.2
$4.0 # 18.8
100.6 = 16.2
18.2 + 21.6
12.5
+ 14.7
90.9 + 21.7
1 16.0
26.1 # 6.5
2000 # 4.5
205 + 5.0
22.0 :
pen-e
154.1
5.4
155.0 + 5.8
1629 =
185.9 + 4.5
00.2 87.7
134.9
= 7.0
141.1 + 8.5
141.3 4.3
130.5 = 16.8
38.0 # 18.2
62.0 $ 9.3
43.0 d: 20.8
dour-h
0.0 4: 0.0
0.0 d: 0.0
0.0 = 0.0
0.0 0.1
5.9 2.5
4.3 = 1.0
46 2 2.0
3
.02 2
-22 0.1
-0.3 $ 6.1
-0.2 0.1
doar-c
1.1 2.6
0.2 0.3
1.2
0.0
0.5
0,0
105.0
105.7
0.2
1.7 0.8
07
0.1
0.0 0.0
0.1
hamne
6.7 8.7
+ 10.4
8.0
0.2
0.7
18.6 + 16:4
+
21.7
home
153..8 0.7
1229 4: 19.7
180.1 de 94.00
131/22 3: 1.0
129.3 1: 0.2
120.5 = 0.1
129.3 4: 0.1
130.5 22 0.2
76.6 20.5
91.1 4: 10.5
96.0 +: 9.0
89.1 11/2
o6-h
0.0 #: 0.0
0.0 + 0.0
0.0 = 0.0
0.0 +: 0.0
0.1 0.1
0.1 = 0.0
0.1 + 0.1
0.1 =: 0.0
0.0 0.0
-0.1 0.0
0.0 = 0.0
-0.1
rele
0.9 8: 1.6
0.4 de 0.5
0.3 m 0.3
0/2
d: 0.2
0.1 0.1
0.1 = 0.1
0.1 1: 0.1
0.1 8: 0.0
0U0 0/0
-0.1 4: 0.0
-0.3 8: 0.0
-0.1 0.0
106.0
3.2
107.8 + 33
107.2 =
108.4 +: 3.5
108.2 0.0
105.7 1.8
109.1 4t 0.7
100.5
30.7 18.8
53. $3.5
+ 3.5
2.5
Aug w/oo
25.5
25.3
24.3
25.5
5.3
16.0
27.2
14.3
0.3
5.0
5.1
5.9
38.6
67.8
66.6
50.0
50.3
58.3
23.4
some improvement but still fell short the original algorithm's performance. Notably, for AutMaze tasks, the hyperparameter adjustments did not improve performance in this domain when compared to just plugging cross-entropy.
For IQL and LB-SAC, tuning the algorithms' hyperparameters with cross-entropy objective helped alleviate inderperformance in Gym-MuJoCo tasks, although random datasets remained common issue. However, this approach did not yield significant. improvements for AntMaze tasks, and only marginally improved average performance in Adroit.
Following the methodology outlined by Kurenkov & Kolesnikov (2022), we investigated whether algorithms utilizing classification offered superior hyperparameter search under uniform policy selection on D4RL tasks, using Expected Online Performance (EOP). The results are prosented in Table 4 under the CE+AT rows. It is evident that for ReBRAC, classification facilitated much better hyperparameter search in the AntMaze domain, marginally improved search for Gym-MuJode and showed slight degradation for Adroit tasks. IQL benefited only in the Adroit domain, while LB-SAC showed a slight improvement in AntMaze tasks.
4.4 What is the impact of classification parameters?
In Farebrother et al. (2024), the authors conducted a limited study on the influence of specific classification hyperparameters, only examining if a/ς = 0.75 was suitable across varying numbers of bins m, using online RL tasks. Drawing inspiration from their work, we selected a set of values: 21, 51, 101, 201, 401, and a set of a/ς values: 0.55, 0.65, 0.75, 0.85. Subsequently, we conducted experiments using all possible pairs of these parameters, while maintaining the parameters of the original algorithms.

---

Published in Transactions on Machine Learning Research (11/2024)
and Table 3 under the CE+CT columns (see also Appendix 1), showcased notable differences in the impact of tuning classification parameters across algorithms. Specifically, ReBRAC exhibited improved performance when classification hyperparameters were fine-tuned, surpassing the original version's efficacy and achieving new state-of-tho-art performance on AntMaze. contrast, IQL and LB-SAC did not consistently benefit from classification parameter luning compared to algorithm-specific adjustments.
Evaluation results, presented in Table Table 2
EOP for this hyperparameter search is also provided in Table 4 under the +CE+CT rows. It's evident that when having good hyperparameters for original algorithm tuning classification parameters yields greater benefits than tuning algorithm-specific hyperparameters, as expected, with the only exception of IQL on Adroit tasks. Additionally, this search converges quickly, with training approxinately three different policies often sufiicigg to achieve near-optimal performance, a highly advantageous property for real-world application of offline RT.
Table 4: Expected Online Performance (Kurenkov & Kolesnikov 2022) under uniform policy selection aggregated over D4RL domains across four training seeds. This demonstrates the sensitivity to the choice of hyperparameters given certain budget for online ovaluation. +CE+AT denotes cross-entropy with tuned algorithm parameters, + +CE+C1 CT denotes cross-entropy with tuned classification parameters, +CE+MT denotes cross-entropy with mixed parameters tuning.
Domain
Algorithm
1 pollcy
2 policies
3 policies
5 policies
10 policies
15 policies
18 policies
20 policies
ReBRAC
62.0 17.1
70.6 1 9.9
73.3 a 5.5
74.8 = 2.1
75.6 = 0.8
75.8 0.6
75.9 0.6
76.0 i: 0.5
ReBRAC+CE+AT
61.1 = 15.4
71. 8.7
71.0 = 488
75.4 = 1.9
76.2 = 1.1
76.6 = 1.0
76.7 + 0.9
76.8 = 0.8
ReBRAC+ CE+
78.4
79.2
0.7
0.5
80.6
80.7
80.7
ReBRAC+CE-MT
62.0
15.5
70.1
10.5
73.3 a 6.6
75.3 L 2.9
76.4 0.9
76.6
0.5
76.7 0.3
JQL
6233 1 9.8
67.6 1 6.0
69.5 = 3.8
70.9 = 1.9
71.6 = 0.7
71.8 = 0.4
71.9 1 0.3
71.9 i: 0.3
Gym-MuloCo
IQL+CE+AT
56.7 +
58.1 3.6
59.3 = 2.8
= 188
61.4 = 0.9
61.7 0.7
61.8
61.8
IQL+CE+CT
58.4 7.2
62.5 1
61.4 = 1.3
68.1
2.6
67.2 1.3
67.8
1.0
67.8 0.9
67.9
IQL+CE+MT
65.2 + 3.6
67.2 + 2.7
68.1 = 2.0
68.9 = 1.2
69.5 0.6
69.7 = 0.4
69.7 + 0.3
LB-SAC
52.5 13.8
50.5
62.0 5.5
+ 3.8
CESAA+CE++T
48.9
11.3
54.4 1.41177 7.0
56.4 1 5.2
38.3
4.1
LB-SAC+E++CT
63.4 3.7
66.1 2.4
66.2 = 1.7
66.9
= 1.0
67.3 = 0.6
LB-SAC+CE+MT
44.4 11.1
50.5 7.9
53.0 + 6-4
2.9
56.3 1.5
56.8
57.0
ReBRAC
67.9 10.0
73.6 7.4
70.1 5.5
78.3
3.4
1.7
80.4 1.1
ReBRAC+ AT
64.8 1 4.9
87.5
1. 3.5
88.7 1. 2.5
89.7 1. 1.5
90.4
0.8
90.7 L 0.5
ReBRAC+CE+CT
$3.8 = 10.3
88.3
4: 4.7
89.4 = 2:2
90.0 = 0.9
90.4
= 0.6
90.6
= 0.4
90.7 0.4
90.7 0.3
MT
63.5 33.0
22.3 79.8
85.8 1. 13.7
89.2
5.2
90.8
1.4
91.2
0.9
91.3 0.7
IQL
21.1 5.4
24.1 11 4.7
25.7 :: 4.2
27.3 =: 3.6
29.5 = 2.7
30.4 =: 2.0
30.7 1.7
30.9 1: 1.5
AntMaze
IQL+CE+AT
15.6
16.1 0.8
16.1 = 0.7
16.7 = 0.7
17.0
0.6
17.2 = 0.5
17.3
17.3
CQL+CE+CT
16.2 +
16.6 0.7
16.8 0.7
17.1
0.7
17.5
0.6
17.7
0.6
17.8 0.5
17.8
IQL+CE+MT
23.3 9.3
28.1 1. 9.9
31.2 1. 9.6
35.1
8.3
39.2
5.1
40.6
3.0
41.0 1 2.3
LB-SAC
10
1.6 1.6 1.6
2.1 1: 1.6
2.8 1.4
LB-SAC+CE+AT
3.9
5.1
6.3 3.4
7.6
2.2
LB-SAC+CE+CI
7.9 1.5
8.7 1.1
9.1 = 0.9
9.4
= 01.
9.6 = D.3
LB-SAC+CE+MT
28
-
6.0 = 3.8
7.6 = 3.1
9.2 = 1.7
9.7 = 1.0
ReBRAC
41.
6.1
2.3
0.9
59.0
59.1
ReBRACC +AT CE++AT
43.9
17.1
52.7 1. 10.3
55.6 1. 6.0
$7.5
2.5
58.5
0.9
58.7
0.6
58.8
0.8
58.9 1. 0.5
ReBRAC+CC++C
56.9 1.7
57.9 1.3 1.3
58.4 = 1.0
58.8
= 0.7
59.2 = D.4
30.3 = 0.3
59.4 10.4:00 0.3
59.4
0.2
ReBRAC+C++MT
17.0 40.8
49.8 12.0
8.1
4.1
LA4
58.0
0.6
08.1
TOL
33.9 2.3
35.2 + 1.7
35.8 + 1.5
36.4 = 1.2
37.1
0.9
37.4 + 0.6
37.5 + 0.6
37.5 + 0.5
Adroit
CE
53.0 3.5
55.0 1. 3.1
56.1
1. 2.7
57.2
1.9
58.1
0.8
58.1
0.4
58.4 0.3
58.4 0.3
CQL+CE+CT
50.1
0.7
0.6
0.4
514
51.5
51.5
IQL+CE+MT
54.2 1 3.1
56.9 1. 2.3
56.7 L. 1.7
57.4
1.0
57.8
0.3
57.8 1. 0.1
57.8 0.1
LB-SAC
15.7 2 14.1
23.4
12.8
27.7
10.5
31.7
6.6
LB-SAC+CE+AT
12.7
17.1
4.3
2.3
LR-SAC+ CT
22.0 2.7
23.4 + 2.6
24.2 + 2.7
25.2
+ 2.6
26.4 + 2.2
LB-SAC+CE+MT
12.5 8.8
17.3 1. 6.5
19.4
1.
4.6
21.1
1
2.6
22.3
1.5
22.8
122
23.0
We present algorithms' performance heatmaps for the parameter grid averaged over domains in Figure with detailed scores per dataset available in Appendix J. Additionally, in Appendix F] we provide further insights by fixing one parameter and averaging over the second. Our analysis indicates that setting m = 101 and //c- = 0.75 may serve as a reasonable starting point and produce results better than the average. Notably,
39 policies for I.B.SACCC+CCT.

---

Published in Transactions on Machine Learning Research (11/2024)
higher number of classes often yields improved performance, suggesting preference for larger values. Regarding 0 //C, our findings suggest that 0.75 remains favorable choice, especially when faced with limited tuning resources. However, the optimal selection of classification parameters heavily depends on the specific algorithm, environment, and dataset characteristics.
ReBRAC Gym-MujoCo
ReBRAC AntMaze
ReBRAC Adreit
90
76.8
77.7
79.3
80.0
76.5
0.85
57.7
87.9
88.3
89.1
91.0
57.4
58.9
54.1
56.3
56.7
50
0.75
0.75
0.75
olt
-79
57
0.65
78.3
77.1
80.3
79.8
77.0
78
0.65
61.1
87.3
88.9
89.6
88.4
0.65
58.0
54.5
56.3
57.3
58.4
-65
-5
0.55-
60
21
51
101
201
401
51
101
201
401
21
51
101
201
401
Number classes
Number of classiss
Number classes
IQL Gym- Gym-MujoCo
0.85
48.3
52.4
55.9
64.0
68.5
165
10.00
0.85
50.0
48.7
50.7
50.0
-1
17.5
0.75
47.5
52.9
59.7
65.9
65.9
60
Scores
0.75
15.8
17.1
16.2
15.6
17.2
17.00
0.75
49.9
48.3
49.2
50.1
51.4
50
6k
16.5
-49
55
0.63
15.7
16.4
16.2
16.6
15.6
16.0
0.55
49.7
52.4
60.9
66.0
67.3
0.55-
15.6
16.5
17.0
15.8
15.6
15.5
0.55
47.9
47.3
50.8
50.7
50.7
101
101
201
401
21
101
201
Number classes
Number of classes
Number classes
LB-SAC Gym-MujoCo
LB-SAC AntMaze
LB-SAC Adroit
0.85
55.8
65.4
66.1
5.8
7.8
9.0
0.85
18.0
21.5
27.9
26
Scores
24
dic
0.75
59.7
65.4
62.0
0.75
0.75-
19.5
24.3
21.2
-2.
0.65
61.6
67.0
67.7
0.65
5.8
7.5
9.8
0.65-
22.0
21.8
21.6
20
18
51
101
201
101
201
51 Number of classes
101
201
Number of classes
Number of classes
Figure 1: Heatmaps for the impact of the classification parameters averaged over the domains. See
Appendix and Appendix for more results.
Farebrother al. (2024) did not investigate the impact of the Unin and Umez choices, nor did they propose method for selecting these when the reward function is unknown. In previous experiments, computed these values by taking the minimum and maximum return across all possible sub-trajectories in the dataset. However, this approach may not be optimal because offline RL algorithms often introduce pessimism into the Q function. Additionally, setting these limits based on extreme values may cause the values on the edges of the support to differ from other values from the model's perspective, potentially impacting the final results.
To evaluate the effect of these parameters, we computed the support size Umaz Umin from the dataset and multiplied this size by a parameter Vezpand- We then extended the support in two ways: Subtracting Vexpond (Umaz Umin)) from Umin (referred to min), and simultancously subtracting from tmin and adding to Umar the term Veapand(Umoz demonstrate that our initial choice of support parameters was sub-optimal and increasing the support range may strongly benefit. Surprisingly, in some cases, reducing the support was beneficial. The both strategy generally yielded better performance.
3/2 (referred to as both The experimental results, shown in Figure 8
Our observations support the assumption about efficiency of cross-entropy for offline RL algorithms with policy regularization compared to other types. Notably, ReBRAC exhibited less scnsitivity to the classification parameters.
4The division by 2 ensures equal bin across the variants.

---

Published in Transactions on Machine Learning Research (11/2024)
Gytt MujoCo
Ad0l
AEDRAC
Ra3RMG,
MARAC both
AERRAC 22ch
BASAC
LO-5AC bath
19-5AC b9ch
18 bel)
-0.05
3.00
0.35
0.3
6.30
-0.3
0.00
2,55
0.0
0.53
-005
0.10
0.65
0.15
0.20
Exparsionisee
Figure 2: Dependency of the algorithms performance on Vexpand values averaged over domains. See Appendix G for labular representation.
4.5 Do MLPs scale better with classification?
In Farebrother et al. 2024 authors emphasized the enhanced scalability of models when regression is replaced with classification, particularly when using Transformers ResNets. In this subsection, we delve into whether similar scalability benefits apply to Multilayer Perceptrons (MLPs), which are commonly utilized in the development of novel RL approaches. Experimental results are presented in Figure 3) Surprisingly, our findings suggest that there is no consistent improvement in terms of scaling when MSE is replaced with cross-entropy.
This discrepancy from the results reported by Farebrother al. 2024) could be attributed to the fundamental architectural differences between MLPs and models like Transformers ResNets. Unlike these latter architectures, vanilla MLPs typically lack residual connections (He et al. 2016), which have been identified as crucial factor in enabling effective scaling with depth. Consequently, the absence of such connect ions in MLPs may limit their ability to capitalize on the benefits offered by classification over regression.
Go
ArtMoue
Adolt
SABRAC
SADRAC RAERAC-CE
A23RAC
MABRACCCE
RAIRAC-CE
Numbero addiiaaal a'kc ayers
Numbor of addioonl crdclayum
Numboc of addk and lec lyyes
Figure 3: Dependency of the algorithms performance on the number of additional layers averaged over domains. See Appendix E for tabular representation.
4.6 Combining the findings
We investigated whether a mixed tuning strategy for hyperparameters could enhance classification performance, building on previous findings. For each algorithm, we tuned the following parameters: from the set Verppndd from the set of {-0.55, 0.05, 0.1) using the both strategy, and three values for one of algorithm-specific parameter: B1 for ReBRAC, JQL T for IQL, and N critics for LB-SAC. For ReBRAC and IQL, the second parameter was held constant across all tasks Jsbsecctio A. a/C was set to 0.75. To enhance readability, we provide detailed per-dataset results a separate tables in Appendix and EOP results Table under the +CE+MT rows (see also Appendix T).
Per-datasel results indicate that the proposed strategy yields better performance in most cases, performing par in others, with the exception of LB-SAC on Gym-MuJoCo. It makes this approach the best choice when no prior knowledge on optimal algorithm-speeific parameters is available.
EOP results reveal that this luning strategy is particularly effective for IQL, especially under a low fine-tuning budget. For ReBRAC, benefits are apparent only under a high fine-tuning budget. Conversely, this strategy did not perform well for LB-SAC. The suboptimal performance of LB-SAC can be attributed to the strong dependence on ensemble size; due computational constraints, we were unable use an ensemble size of 50, which is optimal for many datasets.
9

---

Published in Transactions on Machine Learning Research (11/2024)
To conclude, our experiments show that the best performance with classification can be achieved by tuning algorithm-specifee hyperparaneters in conjunction with classification parameters, particularly the support range and the number of bins.
4.7
Learned Q-functions analysis
ReBBAC antmacc-umaze-22
ReBRAC entmeoe-meedummaagy
ReBRAC antm aeeeaare--eeeeee are-large--plyy-22
80
10asal
80-
70-
(C
70
Q-value
70
-00
60-
60
40-
50
50
200
400
600
800
1000
200
400
600
800
1000
0
200
400
600
800
2000
Training Step / 1000
Training Step/ / 1000
Training Step / 1000
ReBRAC haw.:aammmm----e-eeeereeee.e
ReBRAC aermazzemmdia---ivveee--e
ReBRAC artimaze-aarg-ddiversee
60
70
HMRAAC-C--C
50-
@vvalee
60
Q-veluv
6:50
-50-
30
40
40-
20
200
400
600
800
1000
30
200
400
cbo
600
1000
30-
200
400
600
800
1000
Training Step 1000
Training Step/ / 1000
Training Step 1000
IQL antmaazeumsse-w2
IQL antmaze m-paay.22
IQL entmaac--arge-ppayaee
100-
106
80
12
1244
02-17
60
60-
Q-value
60-
40
40-
60-
20
20-
20-
0-
0-
800
1000
1000
800
1000
Training Step
Training Step/
Training Step 1000
IOL antrnaze-umzze diverse-v2
diverse-v2
100-
100
600-
60-
60-
400-
40-
40
200
0
200
600
600
800
1000
200
800
1000
200
400
600
800
1000
Training Step / 1000
Training Step/
Training Step 10000
Figure 4: Q-value functions behaviour for ReBRAC and IQL on AntMaze tasks. Shaded area demonstrates standard deviation across ten random seeds.
Finally, in Figure 4 we present the dynamics of the Q-function learning for ReBRAC and IQL on AntMaze datasets, as they are of the most interest based on our results. For ReBRAC, modified versions have lower Q-function values on average, but the learning pattern remains similar, except for CE+MT, where the Q-values are notably lower, and the shape of the curve different. However, these differences in the Q-function between CE+MT and CE+AT CE+CT do not result in significant differences in average performance. For IQL, the plots demonstrate that the usage of classification leads to more optimistic Q-functions, with a significant gap between MSE and others. Unfortunately, there is no consistent pattern. We cannot claim that a more optimistic Q-function is the core issue for IQL because, for example, CE+MT is sometimes above CE+CT, but this has no correlation with task performance.
5
Related Work
Prior research beyond the realm of RL has demonstrated the potential performance enhancements associated with replacing regression with classification objectives (Van Den Oord et. al. 2016: Kendall et al.] 2017
10

---

Published in Transactions on Machine Learning Research (11/2024)
Rothe et al. 2018 Rogez et al., 2017). Within the RL domain, some studies have experimented with employing classification objectives as workaround, albeit without conducting comprehensive analyses of this modification (Schrittwieser et al. 2020; Hafner et al. 2023: Hessel et al. 2021 Hansen et al. 2023). Categorical distributional RL (Bellemare et al. 2017) works are also relevant for the considered topic, where the classification is also used, however usage classification instead of regression not a central topic this reseach direction. Additionally, several works in offline RI. have demonstrated the benefits of utilizing classification objectives for various tasks, albeit lacking in-depth analyses of this specific component and its 2024b.
elements (Kumar et al. 2022 Springenberg et al.
To the best of our knowledge, FFarebrother et al. 2024) represent the first and only study to make regression replacement with classification a central research question. The authors compared different methods of converting regression targets into classification targets, providing experimental results across a diverse array of tasks encompassing Atari games, robotics, and natural language processing problems online and offline RL sctups. Their study revealed that HL-Gauss (Imani & White 2018) represents the optimal approach for representing RL regression largets with categorical distributions while this was not the case for the supervised regression tasks (Imani & White 2018). The authors assert that cross-entropy serves as "drop-in' replacement for MSE in RI., leading to more stable training process and enhanced scalability with deep neural network architectures such as Transformers (Vaswani et al. 2017] or ResNets (He et al.) 2016). Our work draws primary inspiration from (Farebrother et al.) 2024)) and aims to provide a more in-depth analysis of this phenomenon in offline RL, which we believe holds significant potential benefits for both offline RL rescarchers and practitioners.
6 Limitations and Future Work
Looking ahead, several avenues for future research emerge. Firstly, one promising direction involves examining how classification affects the performance of offline algorithms in ofllie-too-nling RL setup. Understanding how classification objectives impact the transferability of Iearned policies to online scttings could provide valuable insights into the practical applicability of classification usage. One limitation here, however, is that datasct-specific finite return ranges may restrict policy gencralization when transitioning to online tuning. This could require new design choices to handle potentially expanded return ranges in online scenarios.
Second, our approach applies a relatively straightforward replacement of regression with classification, and future studics could explore more refined adaptations. Although this naive approach fits naturally for ReBRAC and LB-SAC, for IQL we replaced the MSE with cross-entropy only in the Q-network, retaining the expectile loss for the V-function. Expectile loss, as a more general form of MSF, integral to TQT's performance, and investigating whether could be effectively adapted classification-base objective an exciting direction. However, such replacement is mathematically complex and requires additional rescarch. For algorithms like ReBRAC and LB-SAC, which incorporate pessimism in the Q-function, it may also be worth exploring whether discretization should occur before or after the pessimism adjustment, rather than afterward as in our study.
Our findings also offer practical insights for the development of new offline RL algorithms. For example, our results with ReBRAC suggest that classification-baeed objectives could mitigate Q-function divergence, a recurring challenge in off-policy and offline RL (Van Hasselt et al., 2018). Moreover, the guidance we provide hyperparameter choices for classification losses could assist RL practitioners and researchers in fine-tuning these algorithms for better performance if they decide to incorporate classification loss. For instance, a recent modification of ReBRAC by TTrasovv et al. (2024c) is based on our modification of it, leading to near-resolution of the AntMaze domain in the D4RL benchmark suite.
Another theoretically promising direction is leveraging classfication-spccffiss for uncertainty estimation offiin RL. For example, the entropy provided by the Q function could be used incorporate pessimism into offline RI. algorithms.
11

---

Published in Transactions on Machine Learning Research (11/2024)
7 Conclusion
In this study, we explored the impact of integrating classification objectives into offline RL algorithms. Our findings provide nuanced insights into the efficacy of classification in improving offiine RL algorithm performance. Initially, we examined whether classification objectives could be seamlessly integrated into existing algorithms without altering other aspects. While some algorithm with policy regularization (ReBRAC) demonstrated promising results across various tasks, other algorithms with implicit regularization (IQL) and algorithm with Q function regularization (LB-SAC) faced challenges. And the only case when classification have high chances to bring improvement without much effort is the divergence of Q function with MSE loss.
Next, explored the impact of different hyperparameter luning with classification objectives. Notably, we observed performance improvements for ReBRAC, when tuning classification hyperparameters over algorithm- specific oncs. Other results underscore the importance of carcfully selecting both types of hyperparameters: algorithm-sppeciiie and classification-speciifi,, when employing classification.
Furthermore, our investigation into the scalability of MLPs with classification revealed mixed results. Contrary to previous findings with architectures like Transformers and RosNots, we did not observe consistent improvements in scaling when using classification objectives with MLPs. This highlights the importance of considering the architectural nuances of different models when assessing the potential benefits of classification objectives.
In conclusion, our study underscores the need for a muanced understanding of the interplay between algorithm design, task characteristics, and the integration of classification objectives in RL. While classification holds promise in certain contexts, its efficacy is highly dependent on factors such as algorithm design and hyperparameter selection. Future research could further explore these nuances and develop approaches that leverage classification objectives optimally across diverse range of RL tasks and algorithms.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical procipice. Advances in neural information processing systems, 34:29304 22320, 2021.
Dmitriy Akimov, Vladislav Kurenkov, Alexander Nikulin, Denis Tarasov, and Sergey Kolestíkov. Let offline rl flow: Training conservative agents in the latent space of normalizing flows. Xix preprint Xrxiv:221 11090 2022.
Gaon An, Seungyong Moon, Jang- Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in newral information processing systems, 34:7436-7447, 2021.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. In International conference on machine learning, pp. 449-458. PMLR, 2017.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084 15097, 2021.
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang, Latent-variable wevintege-weighted policy optimization for offline rl. orXiv preprint arXiv:2203.0820 2022.
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. orXiv preprinl arXiv:2403.0380
2024.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. ar preprint ar Xiw:2204.00221111 2020.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 1120112-201144 2021.
Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic! estimating uncertainties for offline rl through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35:18267-18281, 2022.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXie preprint orXi:2301.04104, 2023.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16825 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision and paltern recognilion, pp. 770 778, 2016.
Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, and Hado Van Hasselt. Muesli: Combining improvements in policy optimization. In International conference on machine learning, pp. 4214 4226. PMLR, 2021.
Ehsan Imani and Martha White. Improving regression performance with distributional losses. In International conference on machine learning, pp. 2157-2166 PMLR, 2018.
Ehsan Imani, Kai Luedemann, Sam Scholnick-Hughes,, Esraa Elelimy, and Martha White. Investigating the histogram loss in regression. arXiv preprint orXie:2402.13//90 2024.
Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. End-to-end learning of geometry and context for deep stereo regression. In Proceedings of the IEEE international conference on computer vision, pp. 66-75, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:141266980 2014.
llya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXfc:2110.0610 2021.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameterizztion inhibits data-efficient deep reinforcement learning. arXiv preprint 2t1:::0000.1449.. 2020a.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative gccarning for offline reinforce- ment learning. Advances Neural Information Processing Systems, 33:1179 1191- 2020b.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. arXio preprint 2rXiv:2112.04711 2021.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q-learning on diverse multi-task data both scales and generalizes. arXiv preprint 2xXiv:2211.15144 2022.
Vladislav Kurenkov and Sergey Kolesnikov. Showing your offline reinforcement learning work: Online evaluation budget matters. In International Conference on Machine Learning, pp. 11729-11752. PMLR, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0100 2020.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. arxi preprint arXiv:2204.09560 2022.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, Dmitry Akimov, and Sergey Kolesnikov. Q-ensemble for offline rl: Don't scale the ensemble, scale the batch size. arxiu preprint arXim:2211.1 1.1 1.000 2022.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. Xixie preprint wrXiv:2301.33611 2023.
Gregory Rogez, Philippe Weinzaepfel, and Cordelia Schmid. Ler-net: bocaliaaton-ccassiccaiio----s for human pose. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 3433-30 1483-344,, 2017.
Gregory Rogez, Philippe Weinzaepfel, and Cordelia Schmid. +er-net++: Multi-person 2d and 3d pose detection in natural images. IEEE transactions on pattern analysis and machine intelligence, 42(5): 1146- 1161, 2019.
Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision, 126(2):1444155)) 157, 2018
Julian Schrittwieser, Joannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609, 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):3544-35) 359- 2017.
Jost Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Bechtle, Steven Kapturowski, Roland Hafner, et al. Offline actor-critic reinforcement learning scales to large models. arXiv preprint arXiv:2402.0550 2024.
Lawrence Stewart, Francis Bach, Quentin Berthet, and Phin-Philippp Vert. Regression as classification: Influence of task formulation on neural network features. In International Conference on Artificial Intelligence and Slatislics, pp. 11563-111 11582. PMLR, 2023.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024a.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-orient orienned deep offline reinforcement learning library. Advances in Newral Information Processing Systems, 36, 2024b.
Denis Tarasov, Anja Surina, and Caglar Gulcchre. The role of deep learning regularizations actors in oflline rl. arXiv preprinl Stl::2409076000 2024c.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukeuoglu. Pixel recurrent neural networks. In International conference on machine learning, pp. 1747-1756. PMLR, 2016.
Hado Van Hasselt, Yolam Doron, Florian Strub, Matteo Hessel, Nicolas Somnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv: 02648, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in newral information processing systems, 30, 2017.
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjic Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. Advances in Neural Information Processing Systems, 35:23851- 23866, 2022.
Shihao Zhang, Linlin Yang, Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving deep regression with ordinal entropy. arXio preprint arXi::20010080 2023.
14

---

Published in Transactions on Machine Learning Research (11/2024)
A Experimental Details
For results with original algorithms and algorithms hyperparameters search in Table [Talle 2 and Table 3 we conducted & hyperparameler search and selected the best results from the final evaluations for each datasel We used the JAX implementation of ReBRAC from the Clean Offline RT. (CORL) library (Tarasov et al. 2024b) and used the same code template for IQL and LB-SAC. Algorithmic part of IQL implementation is based on the original codebase from Kostrikov et al. (2021) and in case of LB-SAC we have adapted SAC-N implementation from https::///// thub. www.o//wwooooaeeeeoooooo s-
The experiments
conducted on RTX Titan, Quadro RTX 6000 and Quadro RTX 8000.
Our study utilized the v2 version of datasets for Gym-MuJoCo and AntMaze, and v1 for Adroit. The agents were trained for one million steps in all domains and evaluated over ten episodes for Gym-MuJoCo and Adroit and over one hundred episodes for Aate Following Chen al. (2022). AntMaze reward function is multiplied by 100.
For ReBRAC, with O, 0.001, 0.01, 0.1, 0.5 values for Gym-MuJoCo and Adroit. For AntMaze the corresponding ranges are 0.0005, 0.001, 0.002, 0.003 and 0, 0.00011, 0.0005, 0.001. When replacing regression with classification for AntMaze we also set batch size to 1024 and learning rates to 0.001 as it is was done for Gym-MuJoCo in original algorithm which slightly improves the performance.
fine-tuned the 81 parameter with 0.001, 0.01, 0.05, 0.1 values and the 32 parameter
For TQL all domains, we selected 3 value from 0.5, 1,3, 6, 10 and IQL T from 0.5, 0.7, 0.9, 0.95.
For LB-SAC we selected the number of crities in the range of 10, 25,50. Note, that for LB-SAC we used sub-optimal parameters of the batch size and learning rate due to the computational constraints.
A.1 Mixed tuning hyperparameters choice
For ReBRAC, fine-tuned the 31 parameter with 0.001, 0.01, 0.05 values and the B3 parameter was set 0.001.
For IQL, we selected IQL from 0.5, 0.7, and set 8 value to 3.
For LB-SAC we selected the number of critics in the range of 2, 10, 25.
15

---

Published in Transactions on Machine Learning Research (11/2024)
B Algorithms Pseudocodes
Listing 1: Util functions used in our implementations in Python JAX.
def trameform_to_proo (target: jax. Array, support: Jaraa signa: float) -> jaz. Array: cdf_ = baxxscipp special .erf ((support - target) (jnp.sqrt (2) signa))
2 cdt_evall__s_eeeeeeeeaddeee [-1] - cdf_evals [0]
bin_probs cdf_evals [1:] odf (vvasss]]
return bin_prabs / (z 1e-6)
# For batch processing
transforn_to.probe = jax vmap (transform_to_prae in_axes=(O, None, None)) )
def transforn_ transforn_toa_probe (probs: jax Array, support: jax. Array) -> jax. Array:
centers (support [:=] cupport(1:])
return jup.aum (probs centers)
For batch processing
lartooom_rroooeeeeeeeeeeeeeeeeee fron_probs jax. vmap (transform http://rroooooor______ in_axes=(O, None)) transform, from_probs jax. vmap (transform_ frtrtorrrrrrooorobbeee (0_ None))
16

---

Published in Transactions on Machine Learning Research (11/2024)
Listing 2: ReBRAC actor and critic updates with integrated cross-entropy Python JAX psendocode.
def update_actort
key: jax. random. random.PANGkey, actor: TrainState, critic: TrainState, batch: [ict, jax Arrayl, beta: float, tau: float, normalize_q: baol,
):
def actor octor_loss__fpppp (parran: jax. Array):
actions actor. (pplaa batch ("staaess
be_penalty ((actions batch("actions")) ** 2) sum (-1) # Predict categorical distributions logits critic. aptp...facrriiicppraaa..a proba softmax (logite, axis=-1)
batch("state actiona)
Convert distributions scalar values
q_values sransfor__fron_rrott (probs, critic. (0)
lmbda =
normalize_q:
15
Inbda jax. lax. wtt.sstpopgrddeeeee (1 / jax. nuw.y. (qlvee) mean ())
16
loss (beta be_penalty Imbda g_values) mean mean()
return loss
18
grads jax. gtad fn) (a)torr new act_rctor = actor
new neu_actor= actor. apply_aralyygiitttt (grads-grads)
replace
terget.param=-optax. wwwremental wpwpaeettaaluuutttteeeeeeeeaeeee parans. actor target_params tau)
)
aeutcritic pritic.replace
parget.params-optax incremental updaaupetdttttetiteeee (nrre params, critic. targett_aaatte tau)
key, new_actor, new_critic
update_critice
key: jax random. raNcom. actor: TrainState, critic: OriticTrainState,
batch: Dict (str, jax Array], gamma: float, beta: float, policy_noise: float, noise_clip: float.
32 ):
33
key, actions_key jax. randon split
34
next next_actions actor apply_tte (actor. target_ batch ("next_states"]) noise - jax. numpy jax.numpy clip(
(jax. random. normal (actions_key, key, next_actions. shape) policy_moise]) -noise_clip,
moise_clip.
)
next next_actions jax. be_penalty ((nextaacionss batch ("tton""cccctioos 2) .un (-1)
actions noise, -1, 1)
Predict target categorical distributions logita critic. wwptic.ppply_fii....... tttp:rrateeetticttaattppppoppp probs so (logits, axis=-1)
batch ("text_statee"""" next_actions)
Convert distributions next_q Transfom_fron_probopprr (probs, critic. support), .nin (0)
target scalar values
next_q next_q beta bc_penalty
19
target_g batch["reuarde
(1 batch [" ["dones"]) ganma next_q
50
61
def iastic_loos_faccrittie
jax.Array).
# Predict Q categorical distributions
- tapiic..pptytttccttttt
batch ("states" batch ["actions"
# Convert target Q values to categorical distribution
target_probs transfors_to_prant pro.prrettttretggg
(target_q, Pritic.auppore critic.sigma
# Compute CE loss
loss = softmax crtrrre (logitarg, labele=target_pron [None, ....) mean (1)
sun (0)
loss
loa8, grads jax and_ue_anddgraae (critic__c_ (critic params)
critic critic. apply.ppllly.... pooolyggaadeete (grade-grads) return key, nev_critic
17

---

Published in Transactions on Machine Learning Research (11/2024)
Listing 3: IQL actor, V and Q updates with integrated cross-entropy Python JAX pscudocode.
def update_actor(
key: jax. random. rankem.
PRNGKey, actor: TrainState, critic: CriticTrainState, value: TrainState,
batch: [str, Anyl, temperature: float,
):
key, randon
randon aplit 2)
value. wwtpsppppyy.tlllpppp parame, batch ("states=))
Predict categorical distribations
logital, logite2 critic poply.fn(critiee target batch (Mstae"")t batch (" ('actions
"])
probal, probe2 softmax (Logitsl, axis--1), softmar (logits2, axis=-1)
Convert distributions scalar values
q1, q2 transform_fron_proo (probst,, (probsl, critic.support fransorr__fro_prro (probs2, critic. support))
13
jup.minimum (qt, q2)
14
exp_a jap. v) temperature)
exp_a jap. cxpppep -100.0, 100.0)
def actor_loss, faaactor_ppaaane.
dist - actor htt.:eapppyyttctto..rra.aaa batch["states") training-True, rngs- ( d'rooutt
eps 1e-6
1og_probs dist log_ prob (jax. numpy clip (batch(""ctioons (" -1 eps, 1 epa))
actor_loss - -exp__ log_probs) mean ()
return
loss
loss, grads jax. value and and loss_fn) (actor params)
new_ actor accor.applyeeee. return key, nev_actor
(grads
def update_v( key: jax www.raannddnaaa.ca..e critic: GriticTrainState, value: TrainState, batch: [str, Anyl, expectile: float
30
31 ):
Predict Q categorical distributions
logitsl, logits2 Peitic.apply.. in (critic. target_ (atch("sttte="), batch ["actions "})
probsi, probs2 - nn. softmax (logitsi, axis=-1), an softmar axis=-1) a Convert distributions to scalar values
qi. q2 transform_frost critic. support)
(probsl, critic. support), transform troeporrrrroo_prbbee (probs2,
q jap jnp.miinimum(q? q2)
def expectile_ @bpectie__lossdiite (diff, expectile=0.8)
weight jap where (diff 0, expectile, expectile)) return weight (diff
def galue_loss_ftt laree__sos________________ (aavuuus_pparas)
v value. value_loss expectile_lose return value_loss
value.apply apply (wwlleaaaaaaarreeeeeeeeeeeeeaeae batch("states "])
45
expectile) mean ()
46
loss, grads = jax value and grad (value_ loss_ fn) (value parans) new value value. aradiengad (grads-grads) return key, nev_value
def update_q(
key: jax. random. PRNCKey, critic: CriticTrainState, target_value: TrainState,
batch: [etr, Anyl, gamma: float,
next_v larget_valds apply fn (target_ value parans, batch ( (nnext__eet =exx_sstates") target_q batch("rewards"] gamna
batch("dones") next_y Convert target values categorical distribution targot_probs laansfor__to_prote wwwpprotoosttt...... (target_q, critic. support, signa).signa)
18

---

Published in Transactions on Machine Learning Research (11/2024)
def critic_ Peitic_loss_it haa.c_.ooo_.acctti..paraaae
Predict categorical distributions
q1, q2 eritic. Compute CE aps.._yeerrrtti..ppr.....
batch "sttte"sttee""] batch ["sctions"]]
critic_lossi o
watmaaxccosseettteeeeeeeee (logits=ql, labels=target..pro [None, ...]) . mean au).
67
eritic_loss2
(logits=q2, labele=target, [None,
....) . mean a1)
critic_loss critic_lose critic_loss2
return critic_loss
loss, grads - jax value, value_and__eaaeeeae
fn) (critic.ppl (critic.params) (gradssgaaa))
nev_critic aritic.applis return key, nev_critic
Listing 4: LB-SAC actor and critic updates with integrated cross-entropy Python JAX pseudocode. def (
key: random PRNCKey, batch: [ict jax. Array]
TrainState, critic: TrainState, alpha: TrainState,
def actor,
actor_loss_fe los___os_____ forr_loos__faa_e__ (aae__tt(cttrr_prraaa)
actions_dist actor apply__ (actor_paramms batch ("stateet)) actions, actions_logp actions sisttssddisttteeeeeeeeeessssie sestone.dstt.samplleddddddd aample_ (seeedekkk (seed=key) # Predict Q categorical distributions
logits critic. apply_fu(eritiee.pp probs mn. softtmaxx axis--1)
batch ("statee actions)
10
Convert distributions to scalar values q_values
(probs, support) minoortt).inn)))) cannforra_foo_probbe (probs,
loss (alpha tpppayyttaaalppaapaaaaa return loss
actions_logp sun (-1) s.values).nean
loss, grads = jax value and_grad lactor___oss (oss_fn) (actor. neu_actor actor gradients ggrads- return neu_actor
def update_critict
key: ja.. random...... PRNGKey, actor: TrainState, critic: GriticTrainstate, alpha: TrainState, batch: [ict Array, gamma: float, float,
next_actions_diee next next_actions, aext_actions_loge next_ actioacciiss_dttt sag. (seed-key) # Predict target Q categorical distributione
actor actor.apply.. apply_fn params, batch ("extt""netsstateee"""
26
27
logits
batch (ttett"nettsstatte next aext_actions)
probs = nn woftmar axis=-1)
# Convert distributions to target acalar values
next_q = transform_troo_ppoon (probs,
(probs, critic.support. .nin
next_q next_q ap
next actions. sug (-1)
target_q = "recad""e (1 batch ("oces""" * ganma * next_q
def critic_loss_thhcritii...
# Predict Q categorical distributions
q prrtic..ppy.thhcriii.....
batch ["states",, batch ["actions"
# Convert target values to categorical distribution
target_probs transfors_to_ppotte (ooggto supporr pritic.signa
(target_q,
Compute CE loss softax..offax.......omaaa cactaa_cross_eeeeeee (logits tabele-target.prabe [None, ...]) .su (0)
return loss
loss, grads Jax value_allaaan and_grade (eritiee. fn) (critic, params)
new nev_critic = critic. app.y.ppll......... haply.gradiente. (gransssgrrrsssse (grads=grads) soft_ update (tau=tau)
return nev_critic
19

---

Published in Transactions on Machine Learning Research (11/2024)
C Hyperparameters
C.1 ReBRAC
Table 5: ReBRAC's general hyperparameters.
Parameter
Value
optimizer
Adam Kingma & Ba (2014]
batch size
1024 on Gym-MuJoCo and AntMaze (for classification), 256 otherwise
learning rate (all networks)
1e-3 Gym-MuJoCo and AntMaze (for classification), 3e-1 otherwise
tau (T)
5e-3
hidden dim (all networks)
256
num hidden layers (all networks)
gamma (7)
0.999 on AntMaze, 0.99 otherwise
nonlinearity
ReLU
Table 6: ReBRAC's best hyperparameters.
Task Name
A (MSE)
& (MSE)
9 (CE+AT) 3, (CE+AT)
(CE+CT)
a/C (CE-CT)
(CE+MTT
(CE+MT)
Respont (CE+MT)
0.011
0.1
0.001
0.1
101
0.86
0.001
201
0.05
halkchertsh-medinn
hullehoutah-ospers
0.01
0.01
0.01
0.01
201
0.86
0.01
201
0.06
hulfcheetal medium
0.01
0.1
0.01
0.001
101
0.85
0.01
0.1
halfchortab medium replay
hallchoetah-lu hulloalyyyy
0.001
0.1
0.001
0.001
21
0.76
0.001
201
-0.0
hopper randomn
0.001
0.01
0.001
0.5
201
0.85
0.001
401
0.05
hopper-mediumm
0.001
0.001
0.75
-0.0
hopper expert
0.1
0.001
0.1
0.1
201
0.65
0.05
401
0.1
hopper-ma.linne dimmecpprr
0.1
1..0
0.76
hopper--modiuree merii---el
0.05
0.0
0.75
hopper full replay
0.01
0.01
0.01
0.0
101
0.65
0.01
201
0.05
wailbbrrd
0.05
wallberdd-meeee 2d-medium
0.05
0.1
8.15
01.1
101
0.56
0.00
201
0.1
wallin 2d-expert
0.01
0.5
0.01
0.5
101
0.76
0.01
401
0.05
wallerr 2d-medium--eeee expert
vdd medium roplay
0.05
0.01
0.05
0.0
201
0.65
0.01
401
0.00
wallerr 2d fall-rephay
0.01
0.01
0.001
0.001
201
0.55
0.001
201
0.1
matmass- umaue
0.003
not notmare--mmen minee-ivree-
0.003
0.001
0.003
0.002
201
0.76
0.01
201
0.1
mediium play
0.001
0.0005
0.003
0.0005
201
0.65
0.001
0.05
ontmare-mediing mede-m-d-i--iunn
0.001
0.003
0.002
201
0.85
0.01
antmaaee large play
0.002
0.001
0.003
0.0005
101
0.75
0.01
401
0.1
nutuaso-lazgo--iveee
0.003
pen- buman
0.1
0.5
0.1
0.001
0.85
-0.05
pen-ckonel
0.05
0.5
0.1
0.001
401
0.75
0.06
401
0.05
pen expert
doce-hummus
0.1
0.1
0.1
0.1
201
0.86
0.001
401
-0.05
dove-cloned
0.01
0.1
0.1
0.001
101
0.56
0.06
201
0.1
expert
hammer Inman
0.01
0.5
0.1
0.001
51
0.85
0.05
201
0.05
lammer--lood
1186
0.00
hammer-equrrt
0.5
0.55
rebost-human
0.1
0.01
0.1
0.01
201
0.56
0.06
201
0.05
cloorat-cloted
relorate expert
0.05
0.01
0.05
0.5
0.55
0.06
401
0.05
20

---

Published in Transactions on Machine Learning Research (11/2024)
C.2 IQL
Table 7: IQL's general hyperparameters.
Parameter
Value
oplimizer
Adam
Kingma & Ba 2014)
batch size
256
learning rate (all networks)
3e-4
tau (r)
5e-3
hidden dim (all networks)
256
num hidden layers (all networks)
gamma (7)
0.99
nonlinearity
ReLU
learning rate decay
Cosine
dropout rate
0.1 for Adroit, 0 otherwise
Table 8: IQL's best hyperparameters.
Trik Name
IQr. (MSR)
(MSF)
IQQ. (CE+AT)
(CF+AT)
(CR+CT)
mfC (CE+CT)
1QL (CF+MT)
(CE+MIT)
(CR-MIT
ho Inleheetabbrrnddm Ithhetsb-mmelium
0.85
0.7
6.0
0.7
1.0
401
0.65
0.6
401
0.1
nadehocctab-meee medeiium--xpert
0.5
0.5
0.5
3,0
401
0.75
0.7
401
0.1
hadfchetta medium. replay
6.0
tollehoeteddd ailbeuplyy
hogpet random
0.55
10.0
0.55
6.0
51
0.85
0.9
201
0.05
hoopes-mklium
U.7
0.5
0.7
10.0
51
0.80
0.5
201
0.00
3.0
0.05
hogger medrum expert
heggee-mediim reply
6.0
0.85
401
0.05
heepee-full-reglay
0.7
3.0
0.5
3.0
401
0.55
0.5
401
-0.05
we ronddoo walleral-mms-ium
0.9
0.5
0.55
3.0
21
0.65
0.9
401
walkerr
wa marr2t-m-iiummapprr
0.7
3.0
0.5
6.0
401
0.85
0.5
401
0.1
waller wallerralman www.mmmiii--mmppee.. walkerrd-
0.7
1.0
0.5
6.0
401
0.75
0.7
401
0.05
ast trase umaze diverse
0.9
10.0
0.9
6,0
51
0.75
0.5
201
0.05
aurezase modiums play
0.9
6.0
0.9
1.0
51
0.75
0.9
401
0.1
6.0
0.55
6.0
51
0.75
10.0
auturase largo diverse
6.0
0.95
3.0
201
0.55
0.9
201
0.05
peedummn
0.7
0.5
0.7
1.0
201
0.85
0.8
201
-0.05
pen-tlened
10.0
0.5
0.65
pen-nopert
dove-human dboueclonee
0.85
0.25
201
0.7
1.0
0.9
0.5
21
0.75
0.7
201
0.1
0.9
10.0
0.9
6.0
101
0.05
0.7
201
0.00
10.0
21
-0.05
hazamis-ctone.
21
hazames expert
0.55
0.5
0.55
0.5
401
0.65
0.9
401
0.00
relacate homan
0.7
10.0
0.7
1.0
101
0.75
0.9
401
0.05
cllnateclonedd
1.0
0.65
0.05
relscato-osprt
21

---

Published in Transactions on Machine Learning Research (11/2024)
C.3 LB-SAC
Table 9: LB-SAC's general hyperparamcters.
Parameter
Value
optimizer
Adam
Kingma & Ba 2014
batch size
1024
learning rate (all networks)
6e-4
tau (T)
5e-3
hidden dim (all networks)
256
num hidden layers (all networks)
gammaa (7) nonlinearity
0.99
ReLU
Table 10: LB-SAC's best hyperparameters.
Task Name
N crilkes (MSE)
N ctilics (CE+AT)
(CE+CT) o/C (CE+CT)
N critics (CE-MII)
m (CE+MI))
Merpand (CB++MT)
balfchectab-randoo
hulfeheetals-melle
2
5
101
0.75
10
201
-0.05
balfchoetals-co cxppett
51
halfeheetals-mee peeetaalmmlllm-mnneeee.
5
201
0.85
10
201
-0.05
halfchoetals-nn aee-aaaiim-----eeeee-- rrppay
balfchectals-fule ropllp
2
2
201
0.75
401
00.05
bupper-randee
5
0.85
25
0.05
hopper-meciio
25
201
0.75
25
401
0.1
expee--eperr
0.65
25
0.05
poppee--ediion-one
50
51
0.75
25
201
0.1
bupper-une bupper-mediunte peeeittt-replay
5
0.85
10
201
0.05
perpee-full-rpppeee
51
0.75
10
201
0.05
walkerr2d random
51
0.75
10
201
0.05
walleer 24 medmimm
0.85
10
201
-0.05
walkerrd expert
201
0.65
25
00.05
walker 24 meddmm----peeeeeeeee
0.85
10
0.05
walkerrd meditum replay
5
101
0.85
10
201
0.1
Malker2d-fful fiffful-rrppp
0.85
10
201
0.1
and mutreeee--i unnore
5
5
201
0.65
201
0.1
0.75
0.05
and mutee- meweemeeeeeeeeeee mediimm- play
25
5
101
0.75
401
0.05
0.05
antmone-laagee large- play ant aatmmoo-lrrro-diverrs
101
0.75
25
201
0.1
0.75
25
pen- -ummnn
5
10
51
0.75
10
201
0.05
pen-cloned
51
pen- expeet
25
25
201
0.85
25
201
0.1
door-human
dlooncconee
50
201
0.85
25
201
0.05
door-cxpert
commeer-humast
5
50
101
0.75
201
-0.05
cloaer--doued
51
0.85
10
201
-0.05
temmer-opert
25
25
201
0.85
10
0.05
relocate-luman
2
101
0.65
25
401
0.1
relosste-clnodd
0.85
25
-0.05
relocste-expert
101
0.75
25
401
-0.05
22

---

Published in Transactions on Machine Learning Research (11/2024)
D
Computational Costs
Table 11: Computational costs for all experiments. Note, ReBRAC computational costs are taken from
Tarasov et al. (2024a). The total amount of compute is approximately 23497 hours (979 days).
Algorithm
Number of runs
Approximate hours per run
MeBRAC-MSED luning
2784
0.39
ReBRAC+ CE+ tuning
2784
0.34
CBBRAC-C++CT tuning
2880
0.48
ReBRAC-CE+MTT tuning
2592
0.50
IQL+MSE, tuning
2880
0.3
IQL+CE+AT, tuning
2880
0.26
IQL+CE+CT, tuning
2880
0.27
IQL+CE+MT, luning
2592
0.30
LB-SAC+MSE, luming
720
1.29
I.-SAC+CE++AT, tuming
720
1.61
I.-SAC+CE+CCT, tuning
1296
2.32
LB-ACCCE++MI, tuning
2592
1.56
ReBRAC-MSE, eval
360
0.36
ReBRAC+CE, eval
ReBRAC-CE+AT, eval
144
0.47
ReBRAC-CE+CT, eval
144
0.47
CE+ eval IQL+MSE, eval IQL+CE, eval
144
0.34
144
0.35
144
0.33
TQL+CE+AT, eval IQL+CE+CT, eval IQL+CE+MT, eval LB-SAC+MSE, eval LB-SAC+CE, eval LE eval LB-SAC+CE+CT, eval LB-SAC+CE+MT, eval
144
0.24
144
0.34
144
0.34
144
1.25
144
1.34
144
2.21
1.89
ReBRAC-MSE, depth scale ReBRAC-CE, depth scale 4SL+MSE, depth scale
720
0.61
720
0.38
720
0.35
IQL+CE, depth scale
720
0.38
ReBRAC, Vexpand
1440
0.45
IQL+CE, Vezpond
1440
0.34
LB-SAC+CE, Vexpand
1440
2.53
Sum (w/o ReBRAC MSE)
34032
0.69
23

---

Published in Transactions on Machine Learning Research (11/2024)
E
MLPs Scale
Table 12: Dependency of the algorithms performance on the number of additional layers averaged over domains.
Domain
Algorithm
+0 layers
+1 layer
+2 layers
+3 layers
+4 layers
ReBRAC
80.3
81.4
79.7
79.1
80.1
ReBRAC+CE
80.3
79.2
78.8
78.7
80.0
Gym-MuJoCo
IQL
74.0
72.3
68.1
68.3
70.7
IQL+CE
62.4
56.5
57.7
57.0
53.4
ReBRAC
76.4
75.8
61.2
55.8
52.2
ReBRAC+CE
89.2
88.2
86.4
87.8
81.0
AntMaze
IQL
64.5
52.4
52.0
46.2
49.1
IQL+CE
17.0
18.6
15.0
16.8
18.2
ReBRAC
58.1
59.5
55.0
53.4
52.6
ReBRAC+CE
55.2
55.9
55.3
58.8
56.0
Adroit
IQL
33.0
37.4
35.2
33.3
26.4
IQL+CE
48.9
50.9
50.9
48.7
51.7
F Impact of m and a/5
Table 13: Average performance different domains for fixed value and averaged over a a/Ç.
Domain
Algorithm
21
51
101
201
401
ReBRAC+CE
77.6
78.0
80.2
79.5
76.7
Gym-MuJoCo
IQL+CE
48.4
52.6
58.9
65.4
66.6
LB-SAC+CE
59.0
65.9
65.2
ReBRAC+CE
63.7
87.1
89.4
89.1
89.2
AntMaze
IQL+CE
15.5
16.4
16.8
15.9
16.1
LB-SAC+CE
6.0
8.2
9.3
ReBRAC+CE
56.2
56.9
55.5
57.9
58.0
Adroit
IQL+CE
49.3
47.5
49.8
50.3
50.9
LB-SAC+CE
19.8
22.5
23.5
Average (w/o LB-SAC)
51.7
56.4
58.4
59.6
59.5
Average
47.0
49.6
50.6
24

---

Published in Transactions on Machine Learning Research (11/2024)
Table 14: Average performance on different domains for fixed o/Ç value and averaged over m.
Domain
Algorithm
0.55
0.65
0.75
0.85
ReBRAC+CE
78.7
78.4
78.4
78.0
Gym-MuJoCo
IQL+CE
59.2
58.1
58.3
57.8
LB-SAC+CE
59.0
65.9
65.2
ReBRAC+CE
85.3
83.0
83.8
82.7
AntMaze
IQL+CE
16.1
16.5
16.3
15.7
LB-SAC+CE
6.0
8.2
9.3
ReBRAC+CE
56.8
56.8
57.3
56.6
Adroit
IQL+CE
49.4
49.7
49.7
49.4
LB-SAC+CE
19.8
22.5
23.5
Averago (w/o LB-SAC)
57.5
57.0
57.3
56.7
Average
47.4
48.9
48.6
G
Impact of Umin and Umaz
Table 15: Dependency of the algorithm's performance the Vexpand parameter.
Domain
Algorithm
-0.05
0.0
0.05
0.1
0.2
ReBRAC, both
81.0
79.9
80.4
79.8
80.2
ReBRAC, min
71.2
79.9
80.6
80.7
79.0
Gym-MuJoCo
IQL, both
61.3
62.7
61.1
60.2
61.1
IQL, min
61.4
62.7
58.2
57.7
56.7
LB-SAC, both
59.9
64.0
66.1
67.5
65.6
LB-SAC, min
65.8
64.0
64.7
66.1
64.5
ReBRAC, both
90.1
86.3
89.0
89.3
83.6
ReBRAC, min
90.2
86.3
87.5
89.8
86.4
AntMaze
IQL, both
16.7
16.6
34.8
33.0
22.8
IQL, min
16.9
16.6
16.2
16.2
16.2
LB-SAC, both
3.5
8.1
6.0
6.0
4.25
LB-SAC, min
10.3
7.5
7.2
3.5
ReBRAC, both
56.5
54.3
55.5
59.9
55.2
ReBRAC, min
56.7
54.3
56.6
57.7
56.8
Adroit
IQL, both
49.7
49.2
51.8
51.1
51.0
IQL, min
49.4
49.2
50.9
50.0
47.6
LB-SAC, both
23.5
22.3
24.5
22.8
19.1
LB-SAC, min
21.3
22.3
19.3
18.8
21.0
25

---

Published in Transactions on Machine Learning Research (11/2024)
H
Mixed Tuning Results
Table 16: Average normalized score over the final evaluation and ten (four for LB-SAC) unscen training sceds Gym-MuJoCo tasks. CE+MT denotes cross-entropy with mixed hyperparameterrs tuning.
ReBRAC
IQL
LB-SAC
Task
MSE
CE
CE+MT
MSE
CE
CE+MT
MSE
CE
CE+MT
beer
29.5 = 1.5
13.4 + 0.8
9.2 + 0.6
18.9 + 1.0
1.9 0.0
9.0 + 3.0
28.2 + 1.4
10.0 + 0.3
9.9 0.2
he-m
65.6 = 1.0
59.5
+ 0.7
62.0 - 0.5
19.5
+ 1.1
42.5 = 0.3
16.8 = 0.1
61.5
+ 1.3
56.7
+ 2.1
65.5 + 0.5
be-e
105.9 + 1.7
103.2 + 5.5
104.5 + 3.1
95.8
+ 2.2
92.9 = 0.2
94.4
+ 0.3
103.0 + 1.5
103.9 + 1.0
98.2
1.6
he-me
101.1 + 5.2
103.5 + 4.3
104.4 + 2.7
92.3
+ 2.4
86.5 = 4.0
90.7
+ 3.6
104.5 + 2.4
105.4 + 2.0
103.2
2.0
be-mr
51.0 = 0.8
50.7 + 0.7
32.1 + 5.6
45.2
+ 0.5
38.1 = 1.8
43.3
# 0.2
52.8 + 0.7
55.4 + 0.9
$4.5 + 1.1
he-fr
82.1 = 1.1
83.2 = 1.5
82.2 # 1.7
73.5
0.5
62.6 = 1.1
73.2 # 0.5
79.0 + 2.0
80.7 = 0.3
76.6
0..6:11 9.7
hp-r
8.1 = 2.4
8.1 + 0.9
8.5 = 4.0
6.0 6.0
14.2 2.8
9.5 + 0.5
14.5 + 11.5
8.2 + 2.2
14.8
11.0
hp-m
102.0 = 1.0
08.9 9.4
102.0 0.4
54.8 + 4.4
54.1 2.0
50.1 + 5.8
90.0 + 27.5
7.9 0.8
15.6 + 13.5
hp-e
100.1 = 8.3
107.9 + 4.9
110.5 + 0.4
109.4
1.8
110.5 = 0.5
110.7 + 0.2
1.3 + 0.0
21.6
39.7
12.8
12.8
hp-me
107.0 + 6.4
111.6 + 0.5
106.4 + 8.4
88.5 + 16.5
64.0 + 10.2
79.4 + 29.0
111.3 + 0.3
14.9 + 7.0
9.1 5.5
hp-mr
99..1 = 5.3
98.7 1 3.0
100.2 + 6.4
95.9 1 6.4
71.5 = 10.1
83.7 = 11.2
63.0 + 48.1
66.9 = 42.8
66.7 + 32.7
hp-fr
107.1 = 0.4
108.2 + 0.3
108.5 + 0.4
107.2 0.6
41.9 = 4.8
105.7 # 0.3
107.0 + 0.7
65.6 + 50.2
100.3
2.6
wl-r
18.4 = 1.5
8.8 + 141
7.1 # 5.4
6.4 1 6.3
3.9 2.1
9.3 # 5.7
21.7 + 0.0
20.1 + 2.9
21.7 0.1
wl-m
82.5 # 336
85.1 + 2.7
84.5 + 2.6
83.2 + 1.2
79.9
1.3
80.6 + 4.2
89.3 + 5.3
89.6 + 10.7
95.9
3.3
wle
112.3 + 0.2
112.7 + 0.1
112.3 + 0.3
113.8 + 0.2
108.5
0.2
109.5 + 0.2
114.2 + 0.4
59.8 + 45.0
109.8
1.2
wlome
111.6 + 0.3
111.7 + 0.1
110.1 + 0.2
112.3 + 0.6
92.2 = 6.5
110.9 + 0.3
110.6 + 0.4
73.1
18.1
111.3
2.5
wl-mr
77.3 = 7.9
85.2
+ 6.7
83.3
+ 11.7
82.0
+ 7.9
59.1 = 8.8
83.3 = 3.0
92.6 - 2.7
90.9 + 5.3
81.3
8.9
wl-fr
102.2
1.7
101.8
+e 5.6
100.8
+: 19.5
97.7
4 1.4
85.3
3.3
93.7 + 1.1
102.1 de 1.0
110.4 + 1.9
95.7
4.0
Avg
81.2
80.6
80.4
74.1
61.6
71.3
74.9
57.8
63.6
Table 17: Average normalized score over the final evaluation and ten (four for LB-SAC) unscen training seeds on AntMaze tasks.
ReBRAC
IQL
LB-SAC
Task
MSE
CE
CE+MT
MSE
CE
CE+MT
MSE
CE
CE+MT
1.0
98.0
4.6
18.25
35.8
33.2
61.0 + 13.7
88.3 4:
93.0
89.0 +
72.5
med-p
88.0
89.7 1
73.8
0.2
4
med-d
76.3 4: 13.5
84.8
1 9.3
89.8 1
74.8
0.4
0.9
43.9
7.1
0.0
16.0
Irg-p
60.4 + 26.1
85.9
: 6.3
86.2
1
41.3
7.0
0.0 0.0
11.8
1 1.9
0.0
0.0
0.0
Irg-d
54.4
+ 25.1
87.1 +
80.9
23.7
0.0
0.0
4.3
3.1
0.0 0.0
0.0
Avg
76.8
89.4
88.8
60.8
16.5
39.5
3.0
6.8
11.6
26

---

Published in Transactions on Machine Learning Research (11/2024)
Table 18: Average normalized score over the final evaluation and ten (four for LB-SAC) unseen training seeds on Adroit tasks.
ReBRAC
JQL
LB-SAC
Task
MSE
CE
CE++MT
MSE
CE
CE
MSE
CE
CE+
pen-h
103.5
14.1
102.3 1 10.2
86.01 14.5
21.1 1 16.8
108.2
8.4
106.0 1. 13.3
4.5 2.6
7.1 1. 5.6
6.2 1 8.5
pen-c
91.8 + 21.7
90.3 + 14.2
102.2 # 20.4
13.2
21.6
12.5 +
14.7
100.8 4 11.9
26.1 5.3
20.0 + 4.5
16.8 + 8.5
peave
134.9
0.0
0.5
-0.2
door.c
2.6
0.0
0.0 +: 0.D
0.3
0.3 0.3
6.3 + 1.8
0.0 0.0
0.2 = 0.5
0.0 1 0.0
door-e
104.6 1. 2.4
105.7 1.5
105.7 1. 1.4
105.4
2.0
106.1 L. 0.5
105.8 1. 0.0
95.0 .L 8.6
70.6 4. 33.8
80.6 J. 6.2
hanch
0.2 # 0.2
0.1 = 0.1
2.1 2.7
1.7 = 0.8
1.5 = 0.7
3.8 +: 5.2
0.1 1 0.0
0.0 =: 0.0
0.1 1 0.0
ham-c
6.7 1. 3.7
9.7 10.4
2.9 2.4
0.2 L 0.0
1.0 1. 0.7
0.9 L 0.6
20.2
16.8
13.6 1. 15.4
0.3 1. 0.5
122.9 19.7
130.5 + 11.9
129.5
+
+ 10.5
0.0
0.0
rol-c
1.6
0.4
0.2 +: 0.2
0.1
0.1
+ 0.1
0.0
-0.1
0.0 + 0.0
reke
106.6 4: 3.2
107.8
3,3
108.5 4: 2.2
108.2 1: 0.9
105.7 = 1.8
110.0 +: 0.7
26.7
18.8
5.3 +: 3.5
0.6 1 0.2
Avg w/o
25.5
25.3
24.1
5.3
16.0
27.2
6.3
5.0
2.8
Avg
58.6
57.8
58.0
33.1
50.3
59.1
31.6
20.4
24.0
27

---

Published in Transactions on Machine Learning Research (11/2024)
rliable Metrics
1.1 All domains
1000
PNB MAC CE
INSTAC/CE CECECCEETT
To0ete sTrar with Ocoe
75
Algorihm X
Algorithm
RESRACH-CEHCT
ReBRAC
0:25
REBRACHCEHAT.
RedRaC
PHIRAC+CE-
ReBRAC
0.4
0.5
0.6
0.7
0.9
D43A Nomrllood Score
FIX
Y0
Modien
FOM
Heas
Ost malty
REBRAC-CE-CT
REBRAC-CE-AT
RLBRACICE
3.46
0.90
1.00
0,910
DANL
0,900
5.575 RSS
0.870
5.685
0.37
0.18
0:25
2.00
Algoritim
2.75
(0L1CE191)
Algorithm X
IQL-CD-MI
100
2:90
100 +CT+C
100
IQLICE
5.00
0.5
0.1
0.2
0.3
0.4
0.5
D431 Nomralood Score
POX
Medion
Mean
Optimally
10L+
KL-CE-
106
0.36
0.64
0.72
0.80
0.66
DARL Toamellled Score
0.50
0.65
0.70
0.75
0.30
0:35
0.40
1.00
L3-5AC
L3-SAC-CE
DA-SAC-CEAA 18 CC
Algorithm
Algorithm X
LB-SACCCE-MT
T00C of
2:50
LB SAC
0.25
1B-RACHIE+A7
-B-SAC
SAC+CE
SAC
0.
22
0.4
3.6
08
3.
1.2
0.3
0.3
Median
IOM
my Gep
LBSACCCEHNT
CACY
LBGAC- 4AT-
LBSACICE:
LIGAC.
0.45
0.60
0.75
0.4
B Normazze lzed So0re/ 100
9.43
3.54
0.60
5:43
0.0
0.35
2021 metrics for ReBRAC, IQL, and LB-SAC averaged over all Gym-MuJoCo, AntMaze and Adroit datasets. Ten evaluation seeds are used for ReBRAC and IQL and four seeds for LB-SAC.
Figure 5: rliable Agarwal et al.
28

---

Published in Transactions on Machine Learning Research (11/2024)
1.2
Gym-MuJoCo
100
AND BAC-CE
WaRAA-CCE ReBASE-CE+MT
Tmirta oure wih sie
75
Algoritim
Algarithm
RABRNC
1.30
2.25
REBR
RABRAC
REBRAC+CE
5.00
0.
Namralaad Score
06
1.2
FEX 10
Medan
1014
Mean
Optimal by Gep
RETRACCE-MT CEBRACYCECCT
REBRAC+CE+AT
INBRACICE
ALSRAC
0.953
0.975
1.000
1.025
0.915
DAAL Nome Scare 200
0.65
800
0. 0.608
0.86
0216
0.224
0.232
0.240
3.00
Algorithm
1.75
Algorithm X
IOL+CE-NT.
110
Tource offenn with
3:30
IOL
3.25
10LYCEAAT
IQLACE
3.00
0.0
0.6
1.2
0.3
24AL Narralbadd Score
Nedian
1QM
Meen
Optimality Gap
CE
CETTE
10L
CAAL Normalioed Score /130
2.00
13 SAC
L3-SAC-CE
CT
4.75
18 13-5AC- ICE
Algorith X
Algorithm
LB SAC
0:2
L8 SACYCE
SAC
1:00
02
26
0.2
0.3
6.6
PXX x
Optimality Cap
L8 SACCCCEEIIN INT LB CE- CEAT
LB-SAC-CE LB SAC
0.6
Normallon Score/
Figure 6: rliable (Agaraal et al. 2021 metrics for ReBRAC, TQL, and LB-SAC averaged over Gym-MuJoCo datasets. Ten evaluation seeds are used for ReBRAC and IQL and four seeds for LB-SAC.
29

---

Published in Transactions on Machine Learning Research (11/2024)
1.3 AntMaze
BABAAC
ROTTACCEECCT PAORACHCE4MT
Algortthm
Algoráhm
PABRAC
1.2
5.00
RABRACACE-
RABRAC
0.4
6.5
0.
DERL Nemalond Scam
FIX 99
Medan
Mexn
Optiialiiyy
RET
RUBRCCHCEECT
ReBRAC
DEa, lazd Sorey
10L
IQL+CE
CQLECEECT (QU+CC4MT
Algoritim
Algorithm
IOL+GE-NT
100
COL
10L
1:2
KLLCE+AT
10L
3.00
0.5
0.1
5.2
0.3
0.1
0.5
D4NL Nomaliood Score
PXX Y)
Median
Optimallty
ME
10L CT
XOL-CE
IOL
0.3
0.2
0.4
3.5
0.2
D4RL Normaloed Store 1000
2.6
0.30
0.45
0.50
0.45
0.50
0.75
1.00
18 RAC-EE
1.75
Algorithm
CE-SAC
LB-SAC
2.50
LB CEC
3.25
LE 405
LB-SAC
1.00
00
26
08
1.2
0.3
0.5
Nemreleed
Modias
1QM
Mean
Optimallyy by Dep
LBGACHCE-NT
IB-SAC-CE
SAC
0000 0.00
3.008 0.012
0 000
0.000
5,016 0.024
Narmaatad Score/ 300
0.00
0.04
0.08
0.12
0.86
0.50
0.36
1.05
Figure 7: rliable (Agarwal et al. 2021 metrics for ReBRAC, IQL, and LB-SAC averaged over AntMaze datasets. Ten evaluation seeds are used for ReBRAC and IQL and four seeds for LB-SAC.
30

---

Published in Transactions on Machine Learning Research (11/2024)
1.4 Adroit
Algorttbm
Algorihh
8.50
MESRACH-E-CT
PABRAC
2.25
SsSRACA
RASRACHCENC CEACT
RABRACACE-
RABRAC
0.4
6.5
D0R1 Namalond Scam
FIX 93
Medan
Opriiaality
RET
NUBRCCHCECCT
RARAAE-CE-TT
ReBRAC
balc SOFC/100
10:
IQLECEAAT
Algoritim
(QUICE4MT
Erancer df wrT-c0v
IOL+GE-NT
100
CEL
10L
KLICE+AT
10L
3.00
0.4
0.5
5.6
0.7
0.8
0.6
Nomalood
P)
Median
Optimality
ME
10L CE
XOL-CE
IOL.
1.00
1.12
1.30
1:20
1.00
(SRLL Score
1:16
1:04
1.12
1.20
0.00
0.05
0.10
0.15
1.0
SAC-EE
Algorithm
LB-SAC
2.30
SAC
2.26
LE 4A
LB-SAC
1.00
00
36
08
12
0.1
Median
IOM
Mean
Opt mai by Gap
LIGACACE-NT:
SACHCECCT
CET
LBSACHCE
0.5
0.E
1.0
D-LL Narmalaa Scom 100
2.8
1.3
0.75
0.50
0:35
Figure 8: rliable Agarwal et al. datasets. Ten evaluation seeds are used for ReBRAC and IQL and four seeds for LB-SAC.
2021 metrics for ReBRAC, IQL, and LB-SAC averaged over Adroit
31

---

Published in Transactions on Machine Learning Research (11/2024)
J Classification Parameters Performance Heatmaps
J.1 ReBRAC, Gym-MuJoCo
RuBRAC malfche
ReB RAC wcw.cchhhhhhccoo.oo exput-22
385
12.8
128
13.7
12.5
0.0
32
50
0.35
60.3
58.0
56.5
42.5
40.0
1075
205
075
12.7
111
13.2
12.2
0.9
Scores
57.5
1054
1038
196.
102.7
101.3
201
365
12.2
12.9
13.0
11.9
0.0
52.4
59.5
57.2
25.0
0.65
102.1
105.
125.0 105.5
103.0
302
12.5
0.55
12.8
12.
12.7
0.0
0.0
9.55
58.7
35.3
38.5
56.5
35.0
0.5
107.6
98.9
1947
104.7
101.0
201
401
381
301
261
401
Number clsses
Number dasses
Number
ReBRMC hwwfcheetahhhhooom mediun-eeemmm eeplut-exppett222
ReSRAC holfchectoh medium medyyr
ReBRAC halftheetah fulli rollyy?
102.1
0.07
49.0
49.6
47.2
49.0
48.8
0.85
83.1
82.3
79.3
82.6
80.3
104
102
0.75
47.8
49.8
49.9
51.4
48.6
-50
0.75
83.3
80.9
82.1
81.1
80.7
400
0.63-105.4
103.4
99.6
300
0.07-
49.3
50.2
51.0
51.6
51.4
0.65
82.2
72.4
82.6
81.4
79.5
201.3
103.2
104.6
101.0
0.50-
49.2
49.2
51.6
49.0
48.7
0.55
81.3
83.4
82.4
82.9
80.2
21
121
201
461
21
51
103
2011
401
104
201
401
RHBRAC hoppasenndmm-0
hopoermedum
hoopor expertv2
0.9
10.3
0.0
0.55
100.4
86.0
0.85
100.
1031
199 109n
130.8
110
5.6
0.9
8.0
0.0
Scores
3,75
0.75
110.7
130.5
6.9
9.5
0.0
0.55
96.0
78.4
0.88
109.1
1102 111.0
200
0.35
0.5
204
21
51
101
201
601
201
201
401
51
101
201 classes
401
Number cbsses
Number
D-BRAA heppee.ddddmmccaatto
AuthoAC htwppenmeruummee. replaaa--22222222
100
0.55
111.3
108. 188.1 139 112.1
95
0.35
300.c0
109.5
0.75
111.5
109.
111.5 111 109
2075
58.8
80.8
6.75-0776
1:8.6
205.5 108.6 205.0
0.65
110. 110.2
92.3
05.2
96.2
87.5
20.5
0.55-100
6
108.9 1009 108.
104.0
109.8
200.0
3.50
86.4
55.8
98.6
100.0
83.9
307.75
31
101
201
233
461
21
53
333
201
401
ReBRAC wa ker2d random-v2
ReB RAC walker2d eeduumv22
ReBRAC walkerdd expert12
0,85
49
5.7
7.1
0.0
0.0
0.87-
92.1
84.6
86.0
86.5
85.2
-66
112. 112.4 112.3
312
0.76
5.2
6.1
8.1
0.9
0.0
Scrres
0.70-
82.8
84.2
85.9
84.4
85.0
Scates
0.75
9.65
5.1
6.3
10.5
0.0
0.0
0.61-
83.1
85.1
85.9
85.9
84.9
11177
112.7 1115 109.5
0.56
5.5
6.0
3.4
0.0
0.0
0.50-
84.5
85.7
86.2
85.0
85.2
111.7
112.6 112.8
112.5
111.6
508
22
51
204
201
401
21
31
103
201
401
21
51
10s
20a
401
RUBRAC kerdd medium
ReBRAC walkee moltte/2dmedlum-reee am-replay-2
ReSRAC na warroffl-rgge... ay
0.85
80.0
30
0.25
103.6 193.6 163. 100.
100
310. (1007110 111.3
112.0 112
2313
77.4
79.0
77.4
0.75
131.7 104.3 101.9
Scolts
A1C-110.7
111. 112.1 112 112.0
80.2
78.9
0.85
102.7
100.3 1012 102.7
615-1105 111.3
112.0 122
112
5165
035
201 20.
401
51
103
201
601
11
305
261
401
Number of classes
Number classes
Figure 9: Heatmaps for the impact of the classification parameters on Gyn-MuJoCo datasets for ReBRAC.
32

---

Published in Transactions on Machine Learning Research (11/2024)
J.2 ReBRAC, AntMaze
divirsev22
385
$1.8
98.0
97.8
58.5
58.5
0.85
90.0
94.0
93.5
91.0
96.0
0.45
53.5
81.2
90.8
91.5
90.0
80
275
97.0
97.5
98.2
58.5
$9.0
0.75
65.0
92.5
94.2
96.5
66.5
25
0.72
45.0
81.2
92.2
88.0
91.5
2.65
97.2
97.0
98.5
97.5
97.5
0.63
62.0
94.8
93.0
93.5
68.5
0.45
46.8
85.5
91.8
93.0
90.0
2.5
58.8
98.2
97.8
58.5
96.5
0.50
73.0
95.8
96.2
85.0
94.5
0.33
69.8
85.8
92.2
93.0
90.0
Number of classes
Number of classes
Number of classes
entmere-len pwwp plppe/22
10
9.85
$2.4
74.0
15.4
19.55
89.5
100
0.85
40.5
85.5
62.9
83.2
54.2
0.85
35.2
89.0
79.5
80.5
06.8
60
6.75
25.6
21.
84.0
81.9
50.5
0.75
73.9
85.5
85.9
55.5
60.0
0.75
77.5
85.5
85.0
76.0
88.8
965
42.1
80.0
04.9 87.5
00.1
0.65
$7.4
57.0
86.2
53.2
53.2
0.85
60.5
79.5
79.2
02.8
85.0
50
0.5
12.6
64.6
87.1
91.1
87.0
0.55
62.4
85.9
57.0
15.1
80.0
0.55
68.5
85.0
85.0
04.8
81.8
301
201
401
103
401
Number
Number classes
dasses
Figure 10: Heatmaps for the impact of the classification paramcters on AntMaze datasets for ReBRAC.
J.3 ReBRAC, Adroit
RedBtAC renshumennt
RetRAA penselmeddrr
BeTRAC pensecpertoo))
0.85
41.5
102.1
60.2
87.3
B0.4
105
0.35
97.0
131.9
B3.5
82.7
100
0.25
353.1
154.2 155.1 1532 159.4
133
0.71-
93.6
100.5
01.0
108.8
04.2
100
3.75
83.0
82.1
88.1
93.0
1042
Sooree
0.75
237.6
154.3
153.5 1555 157.6
150
0.65
94.1
99.1
92.6
96.1
0.55
100.2
89.4
93.9
97.5
0.69
151.2
155.
153.3 151.3 155.1
0.53-
07.7
15
8.55
91.5
97.2
93.4
0.95
155.3 154.3 155.1 157.0
142
61
201
4.1
10
241
401
81
10)
701
401
Number of classes
Number of desees
Number dlasses
NLBIAC deor-derei-41
RaEHAC doer-espertvv
Taa
0.85
.0.1
-0.9
40.9
0.85
0.0
02
0.1
3.25
0.85
102.7
105.8
100.7 104.6 105.8
0.6
302
0.75
0.1
01
-04
9.75
0.1
0.0
0.1
05
0.1
5.79
1.00
0.75
105
8
104.8
1053 106.1 106.3
0.1
022
3.45
0.0
0.0
0.0
01
0.0
0.66
123.9 105.1 1049 105.3 105.3
20201
0.55
21
23
401
0.0
0.55
-0.00
F1
201
1.5
221
0.0
401
5,00
0.55
21
1946 1052 105.3
102
A01
500
Number of classes
Number dosses
Number classes
nam Numan
ReBRAC hammer
BeSRAC: hammmessppppprra
0.87
0.1
3.0
0.3
0.2
0.1
2.5
0.01-
0.85
119-6
125.3
96.0
131.7
132.0
232
0.75
0.2
0.3
0.1
0.2
0.5
2.0
Scores
0.75
136.1
129.
1018
134.6
134.1
-23
0.85
6.1
1.0
0.65
132.1
125.0
1154
132.0 194.1
0.59
0.55
10.3
0.5
97.8
130. 131.0 134.6 134.6
761
Namber of classes
Number of clevses
Number of clesses
relocate-hmmnne]
0.83
3.0
-0.0
0.0
0.2
6.1
0.4
0.25
0.2
0.5
0.3
0.2
0.3
0.85
189.7 1094
107 169 100.7
0.75
3.0
0.0
6.1
0.1
0.3
0.75-
0.2
0.6
0.2
0.2
1.0
0.75
120.9
1092
108.
105.8
107.4
203
3.8
0.85
4..9
901
0.65
2.4
0.65
109
1065
199.6
1094
109.7
201-0
5.4
0.5
-0.
0.55
0.55
110.1
11311 110. 1100 108.4
307
51
101
201
401
Number of classes
Number of classes
Number claases
Figure 11: Heatmaps for the impact of the classification parameters Adroit datasets for ReBRAC.
33

---

Published in Transactions on Machine Learning Research (11/2024)
J.4 IQL, Gym-MuJoCo
101 baftteees--mnmmm22
101 betchheteeeccteeo
0.8
22
2.3
2.0
22
23
63.5
95.0
0.75
22
2,3
19
23
23
42.6
43.
43.5
45.0
41.0
92.8
92.0
0.63
22
2.3
2.0
24
23
0.65
42.5
41.9
42.2
43.1
43.6
42.5
0.04
92.6
52.0
92.8
932
93.3
92.2
0.53
2.2
2.3
2.1
24
2.
9.55
42.7
42.4
42.8
43.
43.9
42.0
0.55
92.5
93.1
92.8
93.0
93.6
Number classss
Number dassess
Number classes
E2L halicheeahereddeeee perterz
KLL hibheett wwwpaheemmmmmmggaaeeee.o
IBC helfchetahhe dicegley-z2
--40
9.55
45.5
418
47.9
88.3
91.0
0.85
5.4
5.4
59.1
58.7
40.1
0.85
57.2
58.5
61.4
75.6
74.4
70
9.75
45.7
47.4
87.3
88.5
92.5
Stotes
0.75
5.2
11.7
59.1
59.0
57.0
Scoties
0.75
59.3
60.9
62.1
74.5
72.7
Scones
965
45.9
44.3
30.5 01.5
92.1
0.65
9.7
3.1
59.1
59.3
40.4
0.85
59.2
63.1
64.9
74.2
74.7
0.5
45.9
45.7
87.5
89.7
92.2
0.55
11:1
3.6
55.7
59.5
41.3
10
0.55
58.4
62.2
66.7
74.6
74.9
201
201
401
Number
Number
ICL hogger.annoomvv
ICL hopper medium
12hopper expert-22
2,85
10.0
21.0
17.1
9.4
8.8
0.07-
54.2
56.2
51.2
53.5
$1.9
110.6
130.7
110.
1166
0.75
14.8
15.0
15.5
9.4
8.6
0.70
55.2
53.1
52.1
52.9
53.5
0.78
310.7
130.6
110.7
110.0
110.
13.7
15.0
11.7
9.1
8.6
0.01-
53.3
55.1
53.4
54.
55.1
0.35
110.8
110.5
110.7
110.7 110.6
210.90
0.55
15.5
16.1
11.1
9.4
8.3
0.50
53.2
52.7
53.4
53.5
55.2
92
126.
21
51
204
201
401
21
51
101
201
401
23
55
100
261
401
LoLihoppor mod experti22
10L hopper hospee-eduu--rppiiie
hopperfull byy
0.83
53.3
109.1
99.5
100
0.25
30.7
51.9
100.2
55.2
30.0
26.4
0.79
32.9
102.2
0.63
0.65
35.8
36.5
102.5
0.5
123
201
422
51
10)
201
601
21
101
261
401
Number classss
Number of classes
classes
10Lwalier2d random
walkerrd mediumi22
e2pert
5.3
4.1
0.05
0.35-126.6
109.
215.4
4.3
Scoees
77
6.75-2285
1352
229.3
6.65 226.7
128 108.9 109. 109.6
7.8
0.55
72
0.55
300.0
128
200.0 200.3 109
51
101
201
401
Number of classes
Number of classes
Number of dasses
ICE: wel www.brrg--mmeiummeeeeee....
305.8
100.0
109.1 102.5
305
0.85
42.8
60.2
75.0
0.45
80.3
-00
0.75-
96.4
87.5
98.1
105.1
109.5
0.75
5.5
42.7
87.5
75.5
60
0.75
-00
0.63
91.9
85-4
93.5
109.1
1094
0.65
10.5
44.2
66.8
77.7
0.65
0.57-
92.0
80.7
104.2
109-2
108.5
0.55
69.0
77.4
20
0.55
455
Numbor of cbsses
Number of clasees
Number of dasses
Figure 12: Heatmaps for the impact of the classification parameters Gym-MuJoCo datasets for IQL.
34

---

Published in Transactions on Machine Learning Research (11/2024)
J.5 IQL, AntMaze
101 wwtmwaemedumniiiign...
0.85
42.0
66.0
69.2
47.0
66.5
0.85
0.0
0.0
0.5
0.3
0.2
40.5
50.2
47.5
48.8
47.5
45.
0.75
44.2
542
48.0
46.0
50.2
0.75-
0.5
0.5
0.0
0.2
0.0
0.65
47.5
46.0
602
522
47.2
0.63
46.0
51.8
48.0
47.0
67.5
0.65
0.2
0.2
0.2
0.2
0.0
0.55
45.5
51.2
56.8
69.8
51.0
47.5
0.50
48.0
68.0
65.2
64.5
62.5
0.32
0.0
0.2
0.2
7M
Number dlsses
Number of classes
Number classes
120 untm ammmac-mmdiammmee www.-
10L artomaze large 23122
IZL antaaae fer diversov22
0.25
0.87
0.0
0.5
0.5
0.0
0.0
3.85
0.0
0.0
0.0
0.5
0.0
0.0
0.23
0.75
0.0
0.5
0.2
0.0
0.2
0.g
0.0
5.0
0.0
00
9.0
3.0
0.0
0.55
0.65
0.5
0.5
0.0
0.2
0.20
0.0
0.9
0.00
0:65
0.0
9.0
0.0
0.0
0.0
0.59
3.55
0.0
0.0
0.3
0.9
0.55
0.0
0.0
0.0
0.2
0.0
Number of classen.
Numberd of cladses
Number of classes
Figure 13: Heatmaps for the impact of the classification parameters AntMaze datasets for IQL.
J.6 IQL, Adroit
10L pcn Nunion
10L pch expertvl
314.4
102
121.9 107.4
0.85
16.4
23.6
4.4
2.4
11.7
25
321-2
1162
1M2
1402 140.3
142
0.75
113.0
100.5
00.5.30 105.1
113.6
0.75
24.6
14.
244
7.0
11.3
Sooree
0.75 13133
1257
1386 1358 142.8
231
132
0.65
98.9
103.0
106.2
302.0 108.5
0.65
23.9
11.8
17.3
12.5
28.5
0.65
128-
1221
1366
135.0
139.7
125
0.53
97.4
105.4 100.0 169.1 1094
0.99
13.3
16.2
16.7
16.0
15.3
025
110-
1254 141.3 141.1 139.0
129
10: Number clases
Number cbsses
Number of dlssess
IOL
101 deceesperrol
2.0
0.85
2.8
3.1
4,9
2.9
5.3
0.85
1.4
0.7
0.6
0.5
0.6
0.85
124.7 7 105.1 125.1 99.1 95.2
45
0.25
5.6
3.8
3.4
33
5-3
4.0
0.75
2.1
0.5
0.2
0.6
1.5
0.75
194.1 1047
1052 1012 100.7
302
0.63
3.4
3.2
3.2
33
0.05
1.5
0.6
1.2
0.65
194.7
1051
1929
99.5
102.5
0.53
4.6
2.3
3.5
3.0
25
0.55
1.7
0.55
194.8
105.3
125.4 98.3 95.6
10)
Namber classssss
Number of dlasses
Number clsses
humon
hammer dancd
ICL hammen--ogerrii
0.87
1.7
2.1
1.5
1.6
2.0
0.85
1.3
1.2
1.9
1.0
0.8
2.5
0.85
125
1103 128.8 1304 130.7
123
0.75
1.2
1.6
2.0
1.5
1.2
3.0
0.75
0.5
12
2.1
1.5
1.1
0.75
319-2 1034
124.5
1304 130.6
320
0.83
1.3
255
0.55
2.8
1.
0.65
122.9
107.6
120.7
130.
130.7
115.0
2.0
0.59
0.25
035
120.9 1002 130.2 130. 110.7
503
281
Number of classss
Number of clasees
Number of classes
IOL relscatt human v1
108
0.1
0.25
0.0
8.1
0.85
107.8 105. 195. 107. 107.3
0.30
3.3
0.75
0.1
0.2
04
0.1
0.2
0.75
0.75
1374 107.6 10644 105.8 108.1
205
300
0.65
0.1
0.1
02
0.2
0.3
0.65
0.65
105
9
107.2
107.2 107.1 106.5
4.15
2.1
0.55
0.1
0.1
0.2
0.2
0.10
0.55
0.55
125.0
102-
106.0 105. 107.8
122
201
S1
201
401
Numberr dlsssss
Number of classes
Number claases
Figure 14: Heatmaps for the impact of the classification parameters on Adroit datasets for IQL.
35

---

Published in Transactions on Machine Learning Research (11/2024)
J.7 LB-SAC, Gym- Mym-MuJoCo
LB-SAC hallcheetah-randdma
LB-SAC halfchottah-mddinng mediummm?
expertv2
0.85
50.6
105.0
104.8
103.6
105.9
10.5
Score
103.0
0.75
11.0
10.9
8.9
47.5
-50
9.5
-8
95.0
0.65
44.8
54.7
48.6
9.0
10g Number dasies
201
Number of classes
Number of classes
LB-SAC talicheetat-meduu--gbbboe
LB-SAC puticheetaa--lllrssllal
SAC halfchectah medium export.v2
106.2
106.8
107.5
207.5
0.85
52.8
54.4
57.5
0.85
80.2
81.1
81.6
107.0
scoli
Scome
105.7
0.75
106.5
0.65
106
6
106.7
107.8
106.0
53.1
55.1
57.2
78.8
80.7
81.8
79
201
201
201
Number of classes
Number of clssses
LB-SAC hepper-mediumm22
LB LB-SAC eepperex
BB-AAC hopper-randamm/
10.4
0.85
8.0
7.3
7.0
0.85-
1.4
1.8
10.2
10.00
12
Scores
6 0.75
0.75
0.75
30
9.50
20
9.25
0.65
9.3
9.4
10.0
0.65
30
201
LB-SAC hopper-medum-eepppe..o..
LB-SAC hopper wwdium-replaayyyeeeeee
LB SAC hopper-full-eeeee rullaaly----
0.85
36.5
26.0
17.1
0.85
25
103.4
100.7
0:85
130. 6
110.0
85.6
-80
Scores
Scones
6 0.75
69.6
35.3
18.5
40
80.3
104.3
97.2
0.75
110.5
84,2
65.4
20
0.65
87.2
102.5
76.7
0.65
109.
87.5
107.4
51
101
201
101
201
161
201
classes
Number classes
classes
valker2dérambomm22
mediummu
LB-SAC walkerrd expert-v2
28.1
21.7
90.9
98.8
90.8
0.85
4.8
98.1
111.9
100
$5.0
-80
220
Scores
6 0.75
21.6
21.6
21.6
92.5
90.0
0.63
17.6
0.55
99.4
96.1
94.0
87.5
0.65-
32.9
98.7
109.1
101 Number of classess
201
51
101
201
85.6
161
201
Number of classes
Number classes
LD-SAC we wel.ee/demeliiin-gllll.y.
walkerrd -elleplyll--
0.85
95.9
84.3
110.J
0.85
97
0.85
107.2
108.6
110.9
110
100
109
Scoree
3 0.75
0.75
94.0
97.5
95.2
109.0
90
065
90.5
94.2
112.1
0.65
91.8
96.4
93.3
93
0.65
100.
109.7
110.9
101
201
51
101 Number of classes
201
161 Number of classes
201
Number of classes
Figure 15: Heatmaps for the impact of the classification parameters Gym-MuJoCo datasets for LB-SAC.
36

---

Published in Transactions on Machine Learning Research (11/2024)
J.8 LB-SAC, AntMaze
diverse V
medium pay
35.0
0.85
0.0
0.0
0.0
0.85
0.0
0.0
0.0
0 0.75
30.5
55.0
55.8
50
ScoRS
0.0
0.0
0.0
4.00
0.05
Scotes
0:75
0.0
0.0
0.0
0.000
0.05
0.88
35.0
45.2
58.8
40
0.0
51
101
201
35
51
101
251
9..0
51
151
261
0.30
Number of classes
Number of classes
Number classes
LBB5AC wrtmate-medidddeeeeee
LB-SAC antmare-larg-pppppo
0.30
LB-SAC unimaat-laggeedieeeeee
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
+0.0
0.05
0.05
score
0,0
0.05
9.05
0.85
-0.10
-0.10
51
101 Number classes
-0.10
Number of dasses
Number of cassss
Figure 16: Heatmaps for the impact of the classification parameters on AntMaze datasets for LB-SAC.
37

---

Published in Transactions on Machine Learning Research (11/2024)
J.9 LB-SAC, Adroit
pen-human-vl
pen-clonedvv
pen-expert-v)
10
0.85
90
Scornss
Sconns
9.75
0.75
0.75-
70
0.05
0.85
0.05
71.4
50.2
50
51
161
201
51
101
201
51
101
201
Number classes
Number of classes
Number classes
LB -AC doonclonedivl
LB-SAC door-expert-v3
dumehuman-vl
0,200
0.2
-0.1
9.6
0.85
56.8
64.4
89.0
0.85
-0.3
-0.2
-0.3
80
-0.2
-0.250
-011
0.75
51.0
83.1
97.3
-703
0.275
-0.300
0.65
0.65
58.6
68.2
85.2
51
101
201
51
101
261
Number
Number classes
Number classes
LB-SAC hammer-cloned-1
B-SAC hammerchuman-i1
LB-SAC hammerexxpettvv
10.0
0.2
0.3
0.85-
0.1
0.0
0.0
-012
0.10
Scores
0.75
0.08
Scores
0.75
84.4
-0.06
0.04
0.65
7.0
0.3
0.2
94.1
93.9
73.1
51
101
201
Number
Number classes
Number
L8-SAC relectte-export.v1
LB- US-SAC elocate-humaneee
18.SAC relscate-cinedivl
0.050
5.8
0.8
3.3
0.85
-0.1
-0.1
-0.1
0.75
-0.2
-0.1
-0.1
0.125
0.32
T 0.75.
7.7
13.3
6.5
-0.15008
0..
0.0
0.2
-0.200
00.175
0.63-
-0.2
-0.1
.0.1
10.5
51
16L
201
51
161
201
51
101
201
Number of dasses
Number of classes
classes
Figure 17: Heatmaps for the impact of the classification parameters Adroit datasets for LB-SAC.
38


Reviews:
[{"review_id": "66wL8Iw9pb", "reviewer": "Reviewer_DVpN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThis paper presents an enormous amount of experiments, and it is clear that the\nauthors dove deep and rigorously evaluated many properties of the algorithms\nthey tested. In particularly, I commend the authors for their experiments\nassessing the influence of $\\sigma/\\zeta$ as well as their treatment of the distribution\nsupport width, with the introduction of the $v_{expand}$ parameter which was\ninteresting. I really appreciated the heatmap plots (e.g. Figure 1), which\neffectively conveyed the message that it is difficult (and maybe generally\nimpossible) to find a single set of classification hyperparameters that will\nwork well across algorithms and domains.\n\n\n## Weaknesses\n\nThe main issue that stood out to me was with regard to the implementation\ndetails of the classification-based algorithms that were tested.\nIt would&rsquo;ve been great if pseudocode could&rsquo;ve been provided for the\nclassification versions of ReBRAC, IQL, and LB-SAC that the authors tested.\nI perused the source code, and I am not convinced that the implementations are\ntruly proper adaptations of these algorithms: in particular, as actor-critic\nalgorithms, the actor networks are still trained by regression, and in some\ncases, there is a sort of critic baseline that is trained by regression. For\ninstance, in the IQL adaptation, the $V$ function is still trained with\nexpectile regression (as in the original IQL)&#x2014;this function is then used to\nupdate the $Q$ function via cross-entropy. So, altogether, I am not convinced\nthat the results observed in the experiments are fairly judging the influence of\nclassification losses for scaling RL. Similarly, since Farebrother et al.\nconsidered only value-based methods, it is not clear to me that the discrepancy\nin consistency/performance observed in this work is due to the offline nature of\nthe problem, or due to the adaptation of actor-critic algorithms.\n\nMoreover, the work of Farebrother et al. never claims that classification is\n&ldquo;plug-and-play&rdquo; (that I could find, at least). They claim that HL-Gauss is a\n&ldquo;drop-in replacement&rdquo; for MSE, which it is (at least for value-based methods):\nthey simply replace the loss and change the network architecture, but I didn&rsquo;t\ninterpret their work to suggest that this works without hyperparameter tuning.\n\nFurthermore, I believe the study could have been motivated better. In\nparticular, what is it about offline RL in particular that required a more\nthorough examination of classification-based losses?\n\nMore specific details and minor issues are listed in the &ldquo;requested changes&rdquo;\nsection of the review.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Following the recent results of Farebrother et al. (2024) which establish that\ntraining value functions with a classification-based loss enables superior\nperformance and scaling in RL, this work examines whether the phenomenon holds\nacross a variety of offline RL methods in continuous control. Through a wealth\nof experiments, the authors find that classification losses can be quite\nsensitive to their additional hyperparameters in RL, and overall did not exhibit\nsuperior performance with as much consistency as they expected. The authors\nthoroughly examine additional hyperparameters to tune for the classification\nloss and introduce new parameters that can affect quantization (such as a form\nof 'padding' at the return boundaries), and highlight the difficulty of\nfinding a uniformly-good set of hyperparameters.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.  There is a typo in section 2.1 &#x2014; &ldquo;The objective RL of is to&#x2026;&rdquo; should be\n    &ldquo;The objective of RL is to&#x2026;&rdquo;.\n2.  There is a typo in the IQL section on page 3 &#x2014; &ldquo;We opt for IQL as it the\n    best example&#x2026;&rdquo; is missing a word.\n3.  Based on the motivation for using the cross-entropy loss in RL, I would\n    expect it to benefit the actor (in the case of continuous action spaces) as\n    well. However, your code still uses regression losses for the actor, and I\n    think this should be discussed.\n4.  There should be some pseudocode, even if only in the appendix, to depict how\n    you adapt the ReBRAC method for classification loss. My understanding is you\n    compute the target according to the following steps:\n    \n    1.  Compute $Q(s', a', \\hat\\theta)$ by taking the mean of the $Z(s', a', \\hat\\theta)$;\n    2.  Compute $\\mathcal{T}Q = R(s, a) + \\gamma (Q(s', a', \\hat\\theta) - \\beta(a' - \\hat{a'})^2)$;\n    3.  Construct a target distribution $\\hat{Z} = \\mathsf{HL-Gauss}(\\mathcal{T}Q)$.\n    \n    I skimmed the source code, and this appears to be correct.\n    Now, I&rsquo;d expect the action penalty and $\\beta$ to highly influence the support of\n    the distributions $Z(s, a, \\theta)$ &#x2013; how do you account for this?\n5.  Same as above for IQL. Also, why is it appropriate to train the $V$ function by\n    expectile regression if the goal is to train the action-values by classification?\n6.  Same as above for LB-SAC.\n7.  In the LB-SAC section (in section 3), the estimate of the minimum $Q$-value\n    under a Gaussian distribution ensemble isn&rsquo;t clearly relevant. Does this\n    pertain specifically to the classification loss? I suspect not, since the\n    same equation is given in the LB-SAC paper, so I would recommend removing\n    this here because it&rsquo;s unclear what the takeaway is.\n8.  There is a typo in section 4.1 &#x2014; &ldquo;When exploring parameter search in, we\n    utilized&#x2026;&rdquo; is missing something before the comma.\n9.  Using the same hyperparameter grid for ReBRAC and IQL seems unfair due to the\n    concerns I pointed out above. Namely, in ReBRAC, the support of the\n    distribution representations should be fundamentally different. Moreover,\n    these two algorithms do not have the same hyperparameters, so what does it\n    mean to use the same hyperparameter grid for the two?\n10. Why are there no CE+AT+CT columns in Tables 1-3, that is, results where the algorithm\n    parameters *and* the classification parameters are tuned?", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BuCZMMuOnM", "reviewer": "Reviewer_aTU5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n---\n\n\n**1.** The authors show that the classification loss can be plugged easily into three offline RL algorithms. This is a promising result as it will not involve massive modifications to open-source codes and can be directly used by practitioners. The results shown in this paper should be interesting to many researchers in the offline RL field.\n\n\n---\n\n**2.** The authors illustrate that the performance improvements of adopting classification loss are algorithm-dependent. Both when the algorithms or the classification parameters are tuned, certain algorithms can improve substantially while some algorithms are not sensitive to hyper-parameters. The observations in this paper can guide future researchers to tune their methods when applying classification loss to other offline RL algorithms. In some sense, a hope of improvement should not be certainly expected by a simple integration. \n\n---\n\n\n**3.** A previous paper shows that the scalability of value function models can be greatly enhanced when parametrizing the value functions by Transformers or ResNets and using the classification loss. However, this paper gives a negative result on classic multilayer perceptrons.  \n\n---\n\n**Weaknesses**\n\n---\n\n**1.** Three specific offline RL algorithms are analyzed, which are used to illustrate that the classification loss is indeed a \"plug-and-play\" replacement for offline RL algorithms. However, I think the three algorithms are not enough to show that the simple integration always hold. Does there exist some algorithms that cannot be easily \"combined\"? It will be very helpful and make the whole framework more complete if the authors can comment on general offline RL algorithms or provide some counter-examples. In addition, the authors can discuss the reason for selecting these three algorithms as representatives for other offline algorithms.\n\n---\n\n**2.**  The preliminaries and methodology section are not well-presented. Some notations are used without explanations. See my comments below:\n\na. On Page 2, \"Markov Decision Process characterized by a tuple S, A, P, R, γ,\", it would be better to include parentheses.\n\nb. On Page 2, \"to determine a policy (agent) π\", I think the descriptions of agent and policy are not clear and it might be better to drop the agent. \n\nc. On Pages 2-3, The author can put more details when introducing different TD loss in section 2.2., e.g., the HL-Gauss method and the meaning of the parameters in the HL-Gauss method $\\sigma$ and $\\zeta$. Besides, the meaning of the ratio and the reason that 0.75 is a good choice. I understand those results have been shown in a previous paper, it will help the reader to better understand this paper if the background introduction can include more details and be more complete. \n\nd. On Pages 3-4,  \"policy regularization\" and \"computational constraints\" are used without too many details. The authors could comment on which parts of the algorithms that make the computation complicated. \n\n---\n\n**3.** The authors have stated that the random offline datasets are a common issue when stating if something can help the performance. Have the authors done some experiments on the performance of different algorithms when the offline datasets are different? I believe this could be very helpful as the quality of the offline dataset and the differences between the offline dataset and the testing dataset could play a very significant role in the final performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the effectiveness of integrating classification loss with offline RL problems, in particular, three offline RL algorithms are extensively studied, where the corresponding bellman function error functions are modified by cross-entropy losses.\n\nThe authors first show that the integration of the classification loss with offline RL is seamless. Secondly, various tuning schemes are developed as a tool to analyze the performances/sensitivities of the RL algorithms. Empirical results show that by selecting the proper hyper-parameters, solving offline RL problems via classification can outperform classic L2 loss. However, the effectiveness of such modification is not consistent across different offline RL algorithms.\n\nAdditionally, the authors show that the performance cannot be improved by simply enlarging the value function neural network (MLP) size.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Sections 2-3 should include more details about the introduction to offline RL training and corresponding parameters. Section 3 should include some arguments on how and why the three algorithms are selected.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FjOtcCnJ0D", "reviewer": "Reviewer_MQhN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To me, using cross-entropy for value learning is among the most interesting recent trends in Deep RL and the paper presents a valuable study to evaluate this in the context of offline RL. The experiments elucidate the benefits (or lack thereof) across a range of different algorithms and domains of different hyperparameters, in particular those added by introducing the binning for cross-entropy. I highly appreciate the studies including so-far under-explored aspects (beyond considering offline rl instead of online), such as choosing v_min and v_max,  and the scaling analysis for MLPs used in most of RL. \nThe paper is overall clearly written, easy to follow, and provides a valuable and well motivated reference to the (offline) RL community. \n\nThe paper’s key weakness is the lack of rigor applied to establish statistical significance (or lack thereof) in the results: \n- It is a bit unclear to me how many seeds were used. Section 4.1. state 4 seeds were used for final evaluation, but the table captions say ten were used, except for LB-SAC which has only 4. For the final evaluation, I believe 10 should be used for all approaches (also LB-SAC). \n- What exactly is indicated by the +- values in the table? Are those standard deviations or standard errors or something else?\n- Most of the statements in the evaluation consider the performance of a certain algorithm on a certain set of tasks (gym, ant, or adroit) but no values for statistical uncertainty are provided for the “avg” rows in the tables or the entries in the heatmaps. (There are some rliable metrics in the supplement, but they only consider the algorithms across all 3 task sets and only for the findings in tables 1-3) \n\nAdditionally, I found some of the plots and heatmaps hard to read, mostly due to very small font sizes (plots and heatmaps) and poor contrast (heatmaps, white text on yellow tiles, black text on dark blue tiles), and believe this could be easily improved. \n\nAs a final suggestion for improvement, I would highly appreciate a “key takeaways” or “best practices” section in the conclusion. The paper provides a lot of experiments and answers all questions stated in the introduction thought setion 4, but I think a concise, explicit recap of the answers to those questions in the conclusion would highly improve the accessibility of the results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the effects of using the cross-entropy loss for value (q-functions) learning in offline RL. For this, it modifies 3 existing, well-known value-based offline RL approaches (ReBRAC, IQL, and LB-SAC) to use this loss for the critic update and systematically evaluates the effects on the d4rl benchmarks (specifically, the standard gym tasks, ant maze, and the adroit hand tasks). Furthermore, it analyses how using the cross-entropy loss affects both the optimal hyperparameters, as well as finding those in the first place.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify / Add metrics that allow assessing the statistical significance for all results and statements (see above)  \n- Rework readability of plots\n\n*Not required but appreciated:* \n- Explicitly answering the 4 questions posed in the introduction in the conclusion, based on the findings of the work.", "broader_impact_concerns": "n.a.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As detailed above, the paper received mixed reviews with regards to the audience criteria. Based on these comments, I suggest that the authors review the paper accordingly, incorporating the comments from all three reviewers.
Decision: Accept (poster)