Mildly Constrained Evaluation Policy for Offline Reinforcement Learning
Authors: Linjie Xu
zhengyao jiang
Jinyu Wang
Lei Song
Jiang Bian
Venue: Accepted by TMLR ()

Abstract: Offline reinforcement learning (RL) methodologies enforce constraints on the policy to adhere closely to the behavior policy, thereby stabilizing value learning and mitigating the selection of out-of-distribution (OOD) actions during test time. Conventional approaches apply identical constraints for both value learning and test time inference. However, our findings indicate that the constraints suitable for value estimation may in fact be excessively restrictive for action selection during test time. To address this issue, we propose a Mildly Constrained Evaluation Policy (MCEP) for test time inference with a more constrained target policy for value estimation. Since the target policy has been adopted in various prior approaches, MCEP can be seamlessly integrated with them as a plug-in. We instantiate MCEP based on TD3BC (Fujimoto & Gu, 2021), AWAC (Nair et al., 2020) and DQL (Wang et al., 2023) algorithms. The empirical results on D4RL MuJoCo locomotion, high-dimensional humanoid and a set of 16 robotic manipulation tasks show that the MCEP brought significant performance improvement on classic offline RL methods and can further improve SOTA methods. The codes are open-sourced at \url{https://github.com/egg-west/MCEP}.

Published in Transactions on Machine Learning Research (05/2024)
Mildly Constrained Evaluation Policy for Offline Reinforcement Learning
Linjie Xu
linjie.u@qmull.coo uk
Queen Mary University of London
Zhengyao Jiang
z.jiang@es.uuc wcw.:mssss..cllcoooo uk
University College London
Jinyu Wang, Lei Song and Jiang Bian Microsoft Research Asia
/wang-jinyu, lei.song, jiang. / @microsoft.com
Reviewed on OpenReview: https: // openreviee..........mmm. net/ formm? id= 7mAROs79Pb
Abstract
Offline reinforcement learning (RL) methodologies enforce constraints the policy to ad- here closcly to the behavior policy, thereby stabilizing value lcarning and mitigating the selection of out-of-distribution (OOD) actions during test time. Conventional approaches apply identical constraints for both value learning and test time inference. However, our findings indicate that the constraints suitable for value estimation may in fact be excessively restrictive for action selection during test time. To address this issue, we propose a Mildly Constrained Fvaluation Policy (MCEP) for test time inference with get policy for value estimation. Since the target policy has been adopted in various prior ap- proaches, MCEP can be seamlessly integrated with them as a plug-in. We instantiate MCEP based on TD3BC (Fujimoto & Gu 2021b, AWAC (Nair et al. 2020 and DQL (Wang et al., 2023) algorithms. The empirical results on D4RL MuJoCo locomotion, high-dimensional humanoid and a set of 16 robotic manipulation tasks show that the MCEP brought signifi- cant performance improvement on classic oflline RL methods and can further improve SOTA methods. The codes are open-sourced at hhttp://ggitie com/ cwm/egg- hamm//ggwwes//ME/ www.wweettttttt..
constrained lar-
1 Introduction
Offline reinforcement learning (RL) extracts policy from data that is pre-collected by unknown policies. This setting does not require interactions with the environment thus it is well-suited for tasks where the interaction is costly risky. Recently, it has been applied to Natural Language Processing (Snell et al.
2022 Sodhi et al.] 2023), e-commerce (Degirmenci & Jones] 2022) and real-world robotics (Kalashnikov
et al. 2021 Rafailov et al. 2021 Kumar et al.) 2022 Shah et al. 2022: Bhateja et al., 2023) etc. Compared to the standard online setting where the policy gets improved via trial and error, learning with a static offline dataset raises novel challenges. One challenge is the distributional shift between the training data and the data encountered during deployment. To attain stable evaluation performance under the distributional shift, the policy is expected to stay close to the behavior policy. Another challenge is the "extrapolation error' (Fujmoto et al. 2019: Kumar al., 2019) that indicates value estimate error unseen state-action pairs or Out-OO-Distribution (OOD) actions. Worsely, this error can be amplified with bootstrapping and cause instability of the training, which is also known as deadly-triad (Van Hasselt, et al. 2018). Majorities of model-free approaches tackle these challenges by either constraining the policy to adhere closely to the
bchavior policy (Wu et al. 2019 Kumar et al. 2019 Fujimoto Gu 2021 Wang et al. (2023) or regularising the Q to pessimistic estimation for OOD actions focus policy constraint methods.
Kumar al. 2020 Lyu et al. 2022). In this work, we

---

Published in Transactions on Machine Learning Research (05/2024)
Policy constraint methods minimize the disparity between the policy distribution and the behavior distribu- tion. Meanwhile, the strength of policy constraints introduces tradeoff between stabilizing value estimates and attaining better inference performance. While various policy constraints have been developed to ad- dress this tradeoff, it remains a common problem for them that an excessively constrained policy enables stable value estimate but degrades the evaluation performance (Kumar et al. 2019 Singh et al. 2022 Yu ct al., 2023). In particular, the unstable value training may pose the exploding gradient problem. Under this limitation, the valid constraint strengths may not support the best exploitation of the learned value function. In other words, the learned value function may imply better solution that the overly contained policy fails to learn (See Figure 2 and more details in Section 5.1). To reveal this, we investigate the strength ranges of stable value learning and of better inference performance. However, the investigation into the latter question is impeded by the existing tradeoff, as it requires tuning the constraint. without influencing the value learning. To conduct this investigation, we circunvent the tradeoff and seek solutions through the learned value function.
The idea of our approach is inspired by (Czarnecki al. 2019), which has shed light. on the potential of distilling student policy that improves over the teacher using the teacher's learned value function. Therefore, we propose to derive an extra evaluation policy from the value function. The evaluation policy does not join the policy evaluation step thus tunning its constraint does not influence value learning. The actor from the actor-critic is now called target policy as it is used only to stabilize the value estimation. With the help of evoluation policy, we empirically investigate the constraint strengths for 1) stabilizing value learning and 2) better evaluation performance. The results find that a milder constraint improves the evaluation performance but may fall beyond the constraint space of stable value estimation. This finding indicates that the optimal evaluation performance may not be found under the tradcoff, especially when stable value learning is the priority. Therefore, we propose to separate the problems of value learning and evaluation performance, instead of solving them by tradeoff. Consequently, we propose novel approach of using a Mildly Constrained Evaluation Policy (MCEP) derived from the value function to avoid solving the abovenmentioned tradeoff and to achieve better evaluation performance. As the taryet policy is commonly used previous approaches, our MCEP can be integrated with them seamlessly.
The contributions of this work are concluded following:
A novel understanding to the policy constraint methods, showing that the learned value function is not well exploited under the tradeoff between value learning and policy performance. With this insight, we propose to separate the problems of value learning and policy performance instead of solving the tradcoff.
We propose to use an extra mildly constrained evaluation policy with a more constrained target policy, to achieve better policy performance and stable value learning simultaneously.
The performance evaluation on D4RL MuJoCo locomotion, high-dimentional humanoid and set of 16 robotic manipulation tasks show that the MCEP obtains significant performance improvement for policy constraint methods. Morcover, a comparison to inference-tine action selection methods and 3 groups of ablation study verifies the significance of the proposed method.
2 Related Work
policy constraint method (or behavior-regularized policy method) (Wu et al.
2019
Kumar al. 2019 Siegel
et al. 2020 Fujimoto & Gu 2021) forces the policy distribution to stay close to the behavior distribution.
Different discrepancy measurements such as KL divergence (Jaques et al. 2019 Wu et al. 2019 reverse KL divergence (Cai et al. 2022) and Maximum Mcan Discrepancy (Kumar et al. 2019) are applied in previous approaches. Fujimoto & Gu 2021) simply adds a behavior-cloning (BC) term to the online RL method Twin
Delayed DDPG (TD3) Fujimoto et al. 20188) and obtains competitive performances in the offline setting, While the above-mentioned methods calculate the divergence from the data, (Wu et al. (022) estimates the density of the behavior distribution using VAE, and thus the divergence can be directly calculated. Except for explicit policy constraints, implicit constraints are achieved by different approaches. E.g. (Zhou et al.
2

---

Published in Transactions on Machine Learning Research (05/2024)
Qxy
mi
ft2
Qtto
1
Q
1++
π
Q
π e
π
R
QRo
Pl
Pl
Restrictive Constrained Policies
Policy Improvement
Evaluation Policy Space
Policy Evaluation
Figure 1: Left: diagram depicts policy trajectories for target policy क and MCEP τ². Right: policy evaluation steps to update Q# and policy improvement stops to update and n².
2021 ensures the output actions stay in support of the data distribution by using a pre-trained conditional VAF (CVAE) decoder that maps latent. actions the behavior distribution. In all previous approaches, the constraints are applied to the learning policy that is queried during policy evaluation (value learning) and is evaluated in the environment during deployment. Our approach does not count on this learning policy for the deployment, instead, is used as a larget policy only for the value learning.
The policy constraint, well-known to efficient to reduce extrapolation errors, can degrade the policy performance when it overly restrictive. (Kumar et al. 2019) reveals a tradeoff between reducing errors in the Q estimate and reducing the suboptimality bias that degrades the evaluation policy. A constraint is designed to create a policy space that ensures the rosulting policy is under the support of the behavior distribution for mitigating bootstrapping error. (Singh et al., 2022] discussed the inefficiency of policy constraints on heteroskedastic dataset where the behavior varies across the state space in highly non non-uniform manner, as the constraint is state-agnostic. A reweighting method is proposed to achieve a state-aware distributional constraint to overcome this problem. Instead of study this well-known trade-off, we propose to separale the problem value learning and policy performance and devise solution of circumventing the tradeoff by using an extra evaluation policy.
There methods that extract evaluation policy from learned Q estimate. One-step RL Brandfon- brener et al. 2021: Li et al.] 2023) first estimates the behavior policy and its Q estimate, which is later used for extracting the evaluation policy. Although its simplicity, one-step RL is found to perform badly in long-horizon problems due to a lack of iterative dynamic programming (Kostrikov et al. 2022). Kostrikov et al. 2022) proposed Implicity Q learning (IQL) that avoids query of OOD actions by learning an upper expectile of the state value distribution. No explicit target policy is modeled during their Q learning. With the learned Q estimate, an evaluation policy extracted using advantage-weighte regression (Wang al. 2018 Peng et al.) 2019). Our approach has similar form of extracting an evaluation policy from a learned Q estimate. However, one-step RL aims to avoid distribution shift and iterative error exploitation during iterative dynamic programming. IQL avoids error exploitation by climinating OOD action queries and aban- doning policy improvement (i.e. the policy is not trained against the Q estimate). Our work instead tries lo address the error exploitation problem and evaluation performance by using policies of different constraint strengths.
3 Background
We model the environment as Markov Decision Process (MDP) R.T. (.)ss)))) y,), where is the state space, A is the action space, R is the reward function, T(s' |8, a) is the transition probability, Po(s) is initial
3

---

Published in Transactions on Machine Learning Research (05/2024)
state distribution and Y is discount factor. In the offline setting, static dataset D3 = {(s, s')} is pre-collected by behavior policy 2g. The goal is to learn policy Ts(s) with the dataset 'D that maximizes the discounted cumulated rewards in the MDP:
6- arg max Beo (trm) (s)
> R(st, az)]
(1)
Next, we introduce the general policy constraint method, where the policy Th and an off-policy Q estimate Qo are updated by iteratively taking policy improvement steps and policy evaluation steps, respectively. The policy evaluation step minimizes the Bellman error:
(o) Re DasaaeDaaaatoooo [(Qo(sr, 0z) - (r 7Q (st, ()+1
(2)
where the 0' is the parameter for delayed-updated target Q network. The value for the next state is caleulated with actions d+41 from the learning policy that is updated through the policy improvement step:
L (6) D.
+ WC((T mo)]
where C' a constraint measuring the discrepancy between the policy distribution Te and the behavior distribution TT. The w (0) is weighting factor. Different kinds of constraints were used such Maximum Mean Discrepancy (MMD), KL divergence, and reverse KL divergence.
4 Method
In this section, we first introduce the generic algorithm that can be integrated with any policy constraint method. Next, introduce three examples based offline RL methods TD3BC, AWAC and DQL. With mildly constrained evaluation policy, name these three instances as TD3BC-MCEP, AWAC-MCEP and DQL-MCEP.
4.1 Offline RL with mildly constrained evaluation policy
The proposed method is designed to overcome the tradeoff between stable value learning and perfor- mant evaluation policy. In previous constrained policy methods, restrictive policy constraint is ap- plied to obtain stable value learning. We retain this benefit but use this policy (actor) के get policy only to obtain stable value learning. To achieve better evaluation performance, we introduce an MCEP T that is updated by taking policy improvement steps with the value function Q+,- Dif- ferent from Res Th does not participate in the policy evaluation procedure. Therefore, a mild policy constraint can be applied, which helps T go further away from the behavior distribution without influ- encing the stability of value learning. We demonstrate the policy spaces and policy trajectories for #e and TT in the l.h.s. diagram of Figure T where T is updated in the wider policy space using Qap The overall algorithm is shown as pseudo-codes (Alg. D.. At each step, the QF the and zz are up- dated iteratively. A policy evaluation step updates Qa by minimizing the TD error (line G ie. the deviation between the approximate Q and its target value. Next, a policy improvement step updates ig. (line 8 These two steps form the actor-critic algo- rithm. After that, T is extracted from the Q. by taking policy improvement step with a policy con- straint that is likely milder than the constraint for te (line g). Many approaches can be taken to ob- tain a milder policy constraint. For example, tuning down the weight factor w" for the policy constraint term or replacing the constraint measurement. with
lar-
Algorithm 1 MCEP Training
1: Hyperparameters:
2: LR aqq, ait, a,E, EMA 7, w and w
3: Initialize: 8, H¹, th and Φ
4: for i=1, 2, N do
5:
B: D
6:
0 x- aq LL, B.) (Equation
7:
0' + (1 n)0 +
+10
8:
1 t & a= V (v); ú, B.) (Equation
3 Φ - Φ - a V Lo ( w B,) (Equation
9:
4

---

Published in Transactions on Machine Learning Research (05/2024)
less restrictive one. Note that the constraint for T3 is necessary (the constraint term should not be dropped) as the Qi has large approximate errors for state-action pairs that are far from the data distribution.
As the evaluation policy T is not involved in the actor-critic updates, one might want to update To after the convergence of the Qa.- An experiment to compare these design options can be found in the Appendix
Section A.6 Algorithm | that simultaneously updates two policies and these updates (line and 9 can be parallelized to achieve little extra training time based on the base algorithm.
4.2 Three Examples: TD3BC-MCEP, AWAC-MCEP and DQL-MCEP
TD3BC with MCEP TD3BC takes a minimalist modification on the online RL algorithm TD3. To keep the learned policy to stay close to the behavior distribution, behavior-cloning term is added to the policy improvement objective. TD3 learns a deterministic policy therefore the behavior cloning is achieved by di- roetly regressing the data actions. For TD3BC-MCEP, the target policy Ry has the same policy improvement objective as TD3BC:
C+ = ((c.)-oll-QQot (s)))) + (a (s))
(4)
where the ā > .., [Qo())))) is a normalizer for Q values with a hyper-parameter ā: The Qo updated
with the policy evaluation step similar to Eq. 2 using Te. The MCEP T² is updated by policy improvement steps with the Q= taking part in. The policy improvement objective function for zz is similar to Eq. 4 but with a higher-value at for the Q-value normalizer X The final objective for T is
(x) = (lea))=l-xx((( (f() + (a (a- (a- (5))
(5)
AWAC with MCEP AWAC (Nair et al. 2020) is an advantage-weighted behavior cloning method, As the target policy imitates the actions from the behavior distribution, it stays close to the behavior distribution during learning. In AWAC-MCEP, the policy evaluation follows the Eq. 2 with the target policy RØ. that
updates with the following objective:
[-exp exp (Fa(s..)) log (alss)
E((,a)n
(6)
where the advantage A(s,a) =
(s)). This objective function solves an advantage advantagg-weighhted maximum likelihood. Note that the gradient will not be passed through the advantage term. As this objective has no policy improvement term, we use the original policy improvement with KL divergence as the policy constraint and construct the following policy improvement objective:
(Ø) = E,,anD, a ((al A(s, a) + X Dxr (Ta()))))))
(7)
BB, (ar a) X log
(8)
where the weighting factor X is a hypper-arameter. Although the Eq. f is derived by solving Eq. S in a
parametric-policy space, the original problem (Eq. g is less restrictive even with À X as the gradient back-propagates through the -A(s, x (s)) term. This difference means that even with xe > 2, the policy constraint for TO could still be more relaxed than the policy constraint for ă.
DQL with MCEP Diffusion Q-Learning (Wang et al. 2023) is one of the SOTA offline RL methods
that applied a highly expressive conditional diffusion model as the policy to handle multimodal behavior distribution. Its policy improvement step is
L* (1b) = E.-D, XQ(( a) + C(Tss #)),
(9)
where C(T #) is a behavior cloning term and À is the Q normalizer, similar to TD3BC. The policy improve- ment step for the evaluation policy has the same manner as the target policy, except for using different constraint rength.
CeE = EEnD,awT [
> = a) + C(Tg, R)).
(10)
5

---

Published in Transactions on Machine Learning Research (05/2024)
6
Toy MDP
(b) ,T
(c) V+.
(d) Vz, arg max Q+
Figure 2: Evaluation of policy constraint method on toy MDP 2a In other figures, the color of
grid represents the state value and arrows indicate the actions from the corresponding policy. 2b shows the optimal value function and one optimal policy. 2c shows constrained policy trained from the above- mentioned offline data, with its value function calculated by V = Q(,, ((s(ss)(())))) The policy does not perform well in the low state-value area but its value function is close to the optimal value function. 2d indicates that an optimal policy is recovered by deriving the greedy policy from the off-policy Q estimate (the critic).
5 Experiments
In this section, we present experiment results aiming to answer the following research questions. RQ1. Does the learned value function imply better solutions than the constrained policy? RQ2. Can the solution implied by the value function achievable under current policy constraint methods? RQ3. How significantly can the MCEP improve the performance? RQ4. How does MCEP perform compared with other action selection methods that also utilize the value function? Additionally, we adopt groups of ablation studies to verify the benefit of an extra evaluation policy and milder constraints.
Environments D4RL (Fu el al. 2020) is an offline RL benchmark consisting of many lask sets. Our exper- iments select 3 versions of MuJoCo locomotion (-v2) datasets: data collected by rolling out medium- performance policy (medium), the replay buffer during training medium--perfrmance policy (medium- replay), 50% 50% mixture of the medium data and expert demonstrations (medium-czpert) cesaammmc.p......... To in- vestigate more challenging high-dimensional tasks, we additionally collect 3 datasets for Humanoid-v2 tasks following the same collecting approach of D4RL: humanoid ceeirondddddiium---.. humanoid- mawiooooddd-diiu--eeee expert-o2. expeeee---iii--....--........ The humanoid-v2 task has an observation space of 376 dimension and an ac- tion space of 17 dimension. This task is not widely used in offline RL research. (Wang et al. 2020 Bhargava (Bhargava al. (2023]
et al. 2023 considers this task but our data independent of theirs. Compared
we do not consider pure expert data but include the medium-replay to study the replay buifer. The statistics of humanoid datasets listed in Table 5 Finally, consider a set of 16 complex Robotic Manipulation tasks from (Hussing et al. 2023 Their dataset-collecting strategy is similar to the locomotion tasks, hence they are named manipulation-menn medium and
in this work.
Evaluation Protocol As the offline RT. training does not depend on the environment, all the reported results (except for the training process visualization) are produced by evaluating the learned policy on the environment where the data is collected. For visualizing the training process, we save the checkpoints of the policy from different training steps and evaluate them in the environment where the data is collected.
5.1 An illustrative example
To investigate the policy constraint under a highly suboptimal dataset, we set up a loy maze MDP that is similar to the one used (Kostrikov et al. 2022). The euvironment is depicted in Figure 2a where the lower left yellow grid is the starting point and the upper left green grid is the terminal state that gives reward of 10. Other grids give no reward. Dark blue indicates un-walkable areas. The action space is defined as direction movements (arrows) and staying where the agent is (filled circles). There is 25% probability that

---

Published in Transactions on Machine Learning Research (05/2024)
lit mr Po
TD3BC hopper-medium
TD38C waller22d-mediu--relay
AWAC hopper-mediuu-reella
ha-m
ho-m
normalized return
wa-mr
20-
wa-me
wa-m
Figure 4: The training process (with standard errors) of TD3BC and
Figure 5: a values in TD3BC for value estimate and test time inference MuJoCo lo- comotion tasks.
AWAC. Left: TD3BC on hopper-medium--2. Middle: TD3BC on
madlterrddmmdium-eppppaaa Right: AWAC on hosper-medium--eplayyy
random action is taken instead of the action from the agent. For the dataset, 99 trajectories collected by a uniformly random agent and 1 trajectory is collected by an expert policy. Fig. 2b shows the optimal value function (colors) and one of the optimal policies.
We trained a constrained policy using Eq. 2 and Eq. N in an actor-critic manner, where the actor is constrained by a KL divergence with a weight factor of Figure 2e shows the value function and the policy. We observe that the learned value function close to the optimal one Figure 2b However, the policy does not make optimal actions in the lower left areas where the state values are relatively low. As the policy improvement objective shows trade-off between the Q and the KL divergence, in low-Q-value arcas, the KL divergence takes the majority for the learning objective, which makes the policy stay closer to the transitions in low-value areas. However, we find that the corresponding value function indicates an optimal policy. In Figure 2d we recover greedy policy underlying the learned value function (Czarnecki et al. 2019) that shows an optimal policy. In conclusion, the constraint might degrade the evaluation performance although the learned value function may indicate better policy.
5.2 Milder constraints potentially improve performance but cause unstable learning
The maze experiment shows that a restrictive constraint might harm the policy performance, which motivates us to deploy milder constraints that potentially better utilize the learned value function. We investigato this question MuJoCo locomotion tasks. Firstly, we relax the policy constraint on TD3BC and AWAC by setting up different hyper-parameter values that control the strengths of the policy constraints. For TD3BC, we a = {1,4, 10} (in a descending order of the constraint strengths). For AWAC, we set x = (1.0, 0.3, 0.1} } (in descending order of the constraint strengths). Finally, We visualize the evaluation performance and the learned Q estimates.
In Figure 4 the left two columns show the training of TD3BC in the hopper-medium--re and walker2d- moddium-replay-a v2. In both domains, we found that using a milder constraint by tuning the a from to 4 improves the evaluation performance, which motivates us expect. better performance with a = 10. See from the normalized return of 10, we do observe higher performances. However, the training is unstable because of the dramatic change in the magnitude of the Q estimates (note the log scale used in the first row). This experiment indicates the tradeoff between the stable Q estimate and the evaluation performance. The rightmost column shows the training of AWAC in hopper-medium-reelly-e performance by relaxing the constraint (A > 1). Although the Q estimate keeps stable during the training in all À values, higher still result in unstable policy performance (the bottom row) and causes the performance crash with 0.1.
we observe higher evaluation
Concluding on all these examples, a milder constraint can potentially improve the performance but may cause unstable Q estimates unstable policy performances.

---

Published in Transactions on Machine Learning Research (05/2024)
Task Name
BC
CQL
IQL
EQL
TD3BC
AWAC
DQL
100%
10%
original
MCEP
original
MCFP
original
MCEP
balfcheetah-m
42.4
43.1
44.0
47.4
46.5
48.7+0.2
55.5++0.4
45.1+0
46.9
49.8+0.2
03.
hopper-m
54.1
56.9
58.5
65
67
16.1+1.2
91.8:
58.9++19
98.1+0.66
81.7+6.6
95.5+2.2
walker2d-m
71
73.3
72.5
80.4
81.8
85.2+0.9
88.8+00.5
79.6+1.5
81.4+1.6
85.5+0.88
75.3+3..6
balfcheetah-m-r
37.8
39.9
45.5
43.2
43.1
4..
50.6+0.2
43.3++0.1
04.9
47-0.2
47.8+t.11
hopper-m-r
22.5
72
95.0
74.2
87.9
55.2110.8
100
64.826.2
101.1:0.2
100.6:0.22
100.910.33
walker2d-m-r
14.4
56.6
77.2
62.7
71.4
50.9-16.1
86.313.2
84.140.66
83.440.8
93.622.55
92.612.1
balfcheetah-m--e
62.3
93.5
91.6
91.2
89.4
87.1
71.523.7
77.642.6
69.513.8
95.740.4
93.410.8
hopper-m-e
52.5
108.9
105.4
110.2
97.3
91.7=10.5
80.1+12.7
62.448.7
84.3116.4
102.143.0
107.7+1.5
walker2d-m-e
107
111.1
108.8
111.1
109.8
110.440.5
111.7+0.3
109.5:10.2
110.110.2
109.5::.1
109.740.0
Average
51.5
72.8
77.6
76.1
77.1
70.0
81.9
68.3
79.9
85
86.2
Table 1: Normalized episode returns DARL benchmark. The results (except for CQL) are means and standard errors from the last step of 5 runs using different random sceds. Performances that are higher than corresponding basclines are bolded and task-wise best performances are underlined.
5.3 The Evaluation policy allows milder constraints under a stable learning
In this section, we systematically study the constraint strengths on the learning stability and the policy performance. For policy constraint method such as TD3BC, only constraint strengths that do not cause unstable value estimate are valid. To reveal the range of valid strengths, we tune the a for TD3BC within $ = (2.5, 5, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100). For each a (a*), we deploy 5 training with different random seeds. In Figure 5] we visualize the unveiled "safe Q estimate" zone, where the constraint strength enables a stable Q estimate for all seeds. The edge of blue area shows the lowest a value that causes Q value explosion. We found that in 4 of the 9 environments, unstable learning doesn't show up with all constraint strength considered. However, in the remaining 5 environments, the valid strengths is relative narrow.
Next, we are interested in the constraint rengths for the policy performance. We investigate it with the help of evaluation policy. We tune the a for the evaluation policy (TD3BC-EP) within S, with a fixed ā 2.5. The orange area in Figure 5 shows the range of a where the learned evaluation policy outperforms the larget policy. Its edge (the orange line) shows the lowest of values where the evaluation policy performance is worse than the target policy.
Note that a weighs the Q term and thus a larger a indicates a less restrictive constraint. Obscrvod from the orange area, we find that in 7 out of the tasks (7 axis where the orange range is not zero), the evaluation policy achieves better performance than the target policy (ā 2.5). In 5 tasks (5 axis where the orange range is larger than the blue one), the evaluation policy allows milder policy constraints which cause unsafe q estimate in TD3BC. In conclusion, evaluation policy allows milder policy constraints for potentially better performance and does not influence the Q estimate.
5.4 Performance evaluation on MuJoCo locomotion and Robotic Manipulation tasks
D4RL MuJoCo Locomotion We compare the proposed method to behavior cloning, classic offline RL baselines AWAC, TD3BC, CQL and IQL, along with SOTA offline RL methods Extreme Q-Learning (EQL) (Garg et al., 2023) and DQL (Wang et al.) 2023). Following (Fujimoto & Gu 2021), each method uses similar hyperparameters for all datasets. The full list of peperrparammters can be found in Section A.
As is shown in Table [] we observe that the MCEP significantly outperforms their corresponding base al- gorithm (labeled "original"). TD3BC-MCEP gains significant progress on all medium and medium-replay datasets. Although the progress is superior, we observe performance degradation on the medium- datasets which indicates an overly relaxed constraint for the evaluation policy. Nevertheless, the TD3BC- MCEP achieves a much better average performance than the original algorithm. We also provide a per- formance comparison between TD3BC and TD3BC-MCEP with their hyperparameters tuned task-wise (Soction A.4), where we find that TD3BC-MCEP outperforms TD3BC in of the 9 tasks. In the AWAC- MCEP, we observe a consistent performance improvement over the original algorithm on most tasks and the average performance outperos the original algorithm significantly. Additionally, evaluation policies
8

---

Published in Transactions on Machine Learning Research (05/2024)
from both TD3BC-MCEP and AWAC-MCEP outperform the CQL, IQL, and EQL while the target policies have relatively mediocre performances. On the SOTA method, DQL, we found that the MCEP obtains further performance improvement although the improvement is not as large as on conventional methods. This difference may caused by an inference-time action selection method DQL uses. i.c. using the learned value function to filter out an action of high approximate value from the policy distribution, which implicitly loose the constraint. We compare the MCEP with inferenco-time action selection methods Section 5.5
Humanoid One of the major challenges for offiine RL is the distributional shift. In high- dimensional environments, this challenge exacerbated as the collected data relatively more limited. To evaluate the proposed method the ability to han- dle these environments, we col- lect 3 datasets from the MoJoCo Humanoid task. Following the naming of D4RL,, we name these datasets as medium, medium-rrelaay and mediuum-epprr.. The details of data collection and the dataset statistics can be found in Section
medium-v2
medium-reptay-v2
medium-expert-v2
4000
5530c mocp
3000
2000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.1
0.6
0.8
1.0
Figure The returns with standard errors during the training on 3 humanoid tasks.
We compare the TD3BC-MCEP with BC, TD3BC, CRR Wang et al. 2020 IQL and the behavior policy. As seen in Figure G TD3BC-MCEP achieves the highest returns in medium and medium-cxpert. Both of these datasets are collected by rolling out the learned online policy. In medium-replay, where the dataset is the replay buffer of the online training, TD3BC-MCEP also achieves superior performance and shows a faster convergence rate than IQL. Based on the results, we conclude that the MCEP significantly improves the performance on the original algorithm for high-dimensional environments.
Robotic Manipulation Robotic manipulation tasks are recognized as complex tasks for offline RL. We took 16 lasks on the KUKA's IIWA robot from the composition suite (usssing et al. 2023). These tasks consist of 4 basic tasks pickplace, push, shelf, trashcan and target objects boz, dumbbell, hollow- boa, plate. We consider Medium and Medium Medium-Replay datasets. On these tasks, we compare TD3BC- MCEP with BC, CRR, IQL and TD3BC. Similar to the locomotion setting, perparaneter for all 16 tasks. The hyperparam- eter details and full results can be found in Sec- tion A.3 The overall results are presented in Fig- ure where we observe that while TD3BC fails to compete with other methods in Medium and fails to outperform IQI. in Medium-RRplay, the MCEP achieves the highest win rates and outperforms all the baselines in both datasets. The results show that the MCEP is able to improve the performance from the base algorithm even in complex domains.
Average win rates in Robotic Manipulation tasks
IOL TD3BC
TD39C MCEP
consider similar hy-
Medium
Medium Replay
Figure Evaluation (with standard errors) on Robotic Manipulation tasks.
5.5
A comparison to Inference- time action selection methods
The inference-time action selection methods (Wang et al. 2020) provides on-the-fly action selection ap- proach by looking into the value function outpuls. As the MCEP also utilizes the learned value function to generate the evaluation policy, one might want to know the performance difference between the MCEP and the inference-time action selection methods. We consider two types of action selection methods. Argmaz: select the action with the highest estimated q value. Softmaz: sampling action with the probability pro- portion to their estimated values. We compared TD3BC and TD3BC-MCEP the humanoid tasks. To
9

---

Published in Transactions on Machine Learning Research (05/2024)
generate action samples, add Gaussian noise to the policy outputs. We consider standard deviation of 00.0 0.05, 0.1] and consider sample size of [20,50] 50, 100]. The best results are visualized in Figure
Average win rates in Humanoid tasks
From Figure we observe that these action selec- tion methods can improve the performance on both TD3BC and TD3EC-MCEP. By comparing the TD3BC with action selection and TD3BC-MCEP without action selection, note that with Argmaz, TD3BC outperforms TD3BC-MCEP on Medium dataset. However, the Softmax does not achieve the same-level performance. In other datasets, even with action selection, TD3BC still fails to com- pete with TD3BC-MCEP. In conclusion, MCEP brought more significant, performance improvement than dnference-tine action selection methods. The full result can be found in the Appendix A.5
3808
ARGIMXX
080
CWP
MCEP
4.06
M.CENARGMMAY
INCEP-WWP
2006
104
Medium--Replay
Medium-Exper
Figure Comparison with Inference-time action- selection methods.
5.6 Ablation Study
In this section, we design 2 groups of ablation studies to investigate the effect of the extra evaluation policy and its constraint strengths. Reported results are averaged on 5 random seeds.
Performance of the extra evaluation policy. Now, we invostigate the perfor- mance of the introdee evaluation pol- icy x² For TD3BC, we set the parame- ter a (2.5, 10.0). A large indicates milder constraint. After that, we train TD3BC-MCEP with = 2.5 and af = 10.0. For AWAC, we trained AWAC with the A {1.0, 0.5) and AWAC-MCEP with Ā 1.0 and X 0.5.
TD3BC and TD3BC-MCEP
AWAC and AWAC-MCEP
o=25
AeL.
0=10
A-d5
Normalized Return
80
3=
200
21.0
A=1,0=05
04.064.75.77
47.604
40-
haflcheetah
hopper
walker2d
haflcheetah
hopper
walker2d
The results are shown in Figure The scores for different datasets are grouped for each domain. By comparing TD3BC of different a values, we found a milder constraint (a 10.0) brought perfor- mance improvement in hopper tasks but degraded the performance in walker2d tasks. The degradation is potentially caused by unstable value es- timates (see experiment at section 5.2). Finally, the evaluation policy (a = 10.0) with target policy of à = 2.5 achieves the best performance in all three tasks. In AWAC, a lower À value brought policy improve- ment in hopper tasks but degraded performances in half-cheetah and walker2d tasks. Finally, an evaluation policy obtains the best performances in all tasks.
Figure 9: Left: TD3BC with 2.5, a 10 and TD3BC-
MCEP with â a. 10. Right: AWAC with À 1.0, X = 0.5 and AWAC-MCEP with = 1.0 and x = 0.5. The standard errors are also plotted.
In conclusion, we observe consistent performance improvement brought by an extra MCEP that circumvents the tradcoff brought by the constraint.
Constraint strengths of the evaluation policy. We set up two groups of ablation experiments to investigate the evaluation policy performance under different constraint strengths. For TD3BC-MCEP, we tune the constraint strength by sctting the Q normalizer hyper- pappor-prramterr a. The target policy is fixed to ă = 2.5. We pick three strengths for evaluation policy a = (1.0, 2.5, 10.0} to create more restrictive, similar, and milder constraints, respectively. For AWAC-MCEP, the target policy has A= 1.0. However, is not straightforward create a similar constraint for the evaluation policy as has different policy improvement objective. We X {0.6, 1.0, 1.4) to show how performance changes with different constraint strengths.
10

---

Published in Transactions on Machine Learning Research (05/2024)
TD3BC-MCEP
AWAC-MCES
The performance improvements over the target policy are shown in Figure [0] For TD3BC-MCEP, a more restrictive con-
04-
D+1
2-14
Normalizen Return
0.70
straint. (a 1.0) for the evaluation causes significant performance drop. With similar constraint (á = a 2.5).
0.00
001
04-
0.53
40.0
-01-
-02
the performance is slightly improved in two domains. When the evaluation pol- iey has a milder constraint (a = 10), sig- nificant performance improvements are observed in all 3 domains. The right column presents the results of AWAC- MCEP. Generally, the performance in hopper tasks keeps increasing with milder constraints (smaller X) while the half-cheetah and walker2d tasks show performances that are enhanced from À 1.4 to À = and similar performances between À = and À = 0.6. It is worth noting that the evaluation policy consistently outperforms the target policy in halfcheetah and hopper domains. On the walker2d task, a strong constraint (A = 1.4) causes a performance degradation.
-03-
-0.6-
-04
halfcheetah
hopper
walker2d
halfcheetah
hopper
walker2d
Figure 10: Left: TD3BC-EP with 1.0, 2.5 and a 10.0. Right: AWAC-EP with À = 1.4, À = 1.0 and À = 0.6.
In conclusion, for both algorithms, we observe that on evaluation policy, a milder constraint obtains higher performance than the target policy while restrictive constraint may harm the performance.
Estimated Q values for the learned evaluation policies To compare the performance of the policies on the learning objective (maximizing the Q values), we visualze Q differences between the policy action and the data action (s, (s)) Q(s, a) in the training data (Figure 14. 15 in Section A.7). We find that both the target policy and the MCEP have larger Q estimations than the behavior actions. Additionally, MCEP generally has higher Q values than the target. policy, indicating that the MCEP able to move further toward large Q values.
6 Conclusion
This work focuses on the policy constraint methods where the constraint addresses the tradooff between value estimate and evaluation performance. We first investigale the constraint strength ranges for stable value estimate and for evaluation performance. Our findings indicate that the learned value function is not well exploited under this tradcoff. Then we propose to separate the problems of value learning and policy performance, and devise simple and general mildly constrained evaluation policy approach. The novel approach circumvents the above-mentioned tradeoff thus achieves stable value learning and policy performance simultaneously. The empirical results on locomotion, humanoid and robotic manipulation tasks show that MCEP can obtain significant performance improvement.
Limitations. Although the MCEP is able to obtain better performance, the evaluation policy requires extra effort in tuning its constraint strength. We suggest starting from the strength of the target policy and trying milder constraints. Note that an effective constraint is still indispensable to avoid too many OOD actions that risk real-world application. Moreover, the performance of the MCEP depends on stable value estimation. Unstable value learning may collapse both the target policy and the evaluation policy. While the target policy may recover its performance by iterative policy improvement and policy evaluation (actor- critic), we observe that the evaluation policy may fail recover its performance from the collapse. Therefore, restrictive constrained target policy that stabilizes the value learning is essential for the proposed method.
11

---

Published in Transactions on Machine Learning Research (05/2024)
References
Prajiwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, and Amy Zhang. Sequence mod- eling is robust contender for offline reinforcement learning. arxie preprint 2Xxv:2305.14550 2023.
Chethan Bhateja, Derek Guo, Dibya Ghosh, Anikait Singh, Manan Tomar, Quan Vuong, Yevgen Chebotar, Sergey Levine, and Aviral Kumar. Robotic offline from internet videos via value-function pre-training. arXiv preprint 2i0: 13041, 2023.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offlin rl without off-policy ovaluation. Advances in neural information processing systems, 34:4933-499 4946, 2021.
Y. Cai, C. Zhang, L. Zhao, W. Shen, X. Zhang, L. Song, J. Bian, T. Qin, and T. Liu. Td3 with reverse kl regularizer for offline reinforcement learning from mixed datasets. In 2022 TEEE International Conference on Data Mining (ICDM), pp. 21-30, Los Alamitos, CA, USA, dec 2022. IEEE Computer Society. doi: 10.1109/ICII /t/////DMM444..22.00000 2022.00012
URL httpt: //doi. www.eccotttteeeeccier......... 1r11://0o//111110000/0444444
Wojeiech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. In The 22nd international conference on artificial intelligence and statistics, pp. 1331-1340. PMLR, 2019.
Soysal Degirmenci and Chris Jones. Benchmarking offiine reinforcement learning algorithms for e-commerce order fraud evaluation. In 3rd Offinne RL Workshop: Offline RL as a "Launchpad", 2022. URL https:
//openreview.co http:e/eeeeeeeoooorr//ppppppooo.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datascts for deep data- driven reinforcement learning, arXie preprint wxXiv22044007211 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to oflline reinforcement learning. Advances in neural information processing systems, 34:20132 201142,, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587 1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations, 2023. URL https:
/openneview..c http://wwwne///ooottddidiisi
Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalnikov, Piotr Stanczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Dulac-Arnold, Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed Kam- yar Seyed Ghasemipour, Sertan Girgin, Olivier Pietquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research frame- work for distributed reinforcement. learning. ar preprinl ar arXiv:2006.000 00979, 2020. URL https: //arxiv. http:/bb//0000 00999
Marcel Hussing, Jorge A Mendez, Anisha Singrodia, Cassandra Kent, and Eric Eaton. Robotic manipulation datasets for offline compositional reinforcement learning. arXiv preprint arXio:2307.07091, 2023.
Natasha Jaques, Asma Ghandcharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off off-policy batch deep reinforcement learning of implicit human preferences in dialog. Xixiv preprint or X 1907 00456, 2019.
12

---

Published in Transactions on Machine Learning Research (05/2024)
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale, arXiv preprint arXiv:2104.08212, 2021.
Ilya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2022. URL
htttp://giihhb. ht/m//kkoorrko//rrrl2222 v2.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offine reinforcement learning with implicit q-learning. In International Conference on Learning Represenlations, 2022. URL. https://openrrevior
http://ppaarrrreeeee..eftoooaooa net/ffrummidd-
68
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-lcarning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Newral Information Processing Systems, 33:1179 1091-119,,1111 2020.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model- free robotic reinforcement learning. In Conference on Robot Learning, pp. 417 428. PMLR, 2022.
Jiachen Li, Edwin Zhang, Ming Yin, Qinxun Bai, Yu-Xiang Wang, and William Yang Wang. Offline re- inforcement learning with closed-form policy improvement operators. In International Conference Machine Learning, pp. 20485 20528. PMLR, 2023.
Jiafci Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-larnning for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Newral Information Processing Systems, 2022. URL https://openreriie: wttp://wwww.........ooooo/ooeeeo
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement
learning with offline datasets. arXiv preprint arXiv:2006.0935 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage Advantage-weighteed regression: Simple and scalable off-policy reinforcement learning. arXiv preprinl or 1rXi::1910000177, 2019.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp. 1154 1168. PMLR, 2021.
Dhruv Shah, Arjun Bhorkar, Hrishit Leen, llya Kostrikov, Nicholas Rhinehart, and Sergey Levine. Offline reinforcement learning for visual navigation. In 6th Annual Conference on Robot Learning, 2022. URL
https:://pperreview wtt.://www........o.ooorraaaaaaa
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior mod- elling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020. URL htttps://openreviee..u httparrwwwweeeeeeeeeooooeerrrrte
Anikait Singh, Aviral Kumar, Quan Vuong, Yevgen Chebotar, and Sergey Levine. Offline rl with realistic datascts: Heteroskedasticity and support constraints. arXiv preprint arXim:2211.01052. 2022.
Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. Xr preprinl arXiv:2206. 11871, 2022.
Paloma Sodhi, Felix Wu, Ethan R Elenberg, Kilian Q Weinberger, and Ryan McDonald. On the effectiveness offline rl for dialogue response generation. In International Conference on Machine Learning, pp. 32088- 32104. PMLR, 2023.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint ar 0264448 2018
13

---

Published in Transactions on Machine Learning Research (05/2024)
Qing Wang, Jiechao Xiong, Lei Han, Peng Sun, Han Liu, and Tong Zhang. Exponentially weighted imita- tion learning for batched historical data. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6291 6300, 2018.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et. al. Critic regularized regression. Advances in Newral Information Processing Systems, 53:7768-77775 2020.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long, Supported policy optimization for offline reinforcement learning. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL httpp:///penrrvieew.ee
Forum/1d=KCXQSHol-1y
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint Xrxiv:1911. 1.111.111.000 11361, 2019.
Lantao Yu, Tianhe Yu, Jiaming Song, Willie Neiswanger, and Stefano Ermon. Offline imitation learning with suboptimal demonstrations via relaxed distribution matching. ar preprint arXiv:2303.025: 2023.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, pp. 1719 1735. PMLR, 2021.
14

---

Published in Transactions on Machine Learning Research (05/2024)
A Appendix
A.1 The implementation details and hyper-parameters for locomotion evaluation
For CQL, we reported the results from the IQL paper (Kostrikov et al. 2022 to show CQL results '-v2' tasks. For IQL, we use the official implementation (Kostrikov] 2022 to obtain a generally similar performance as the ones reported in their paper. Our implementations of TD3BC, TD3BC-MCEP, AWAC, and AWAC-MCEP are based on [Kostrikov] 2022) framework. In all re-implemented iemmllonnttrcdmmppmmennndddd methods, clipped double Q-learning (Fujimoto et al.)
2018 is used. In TD3BC and TD3BC-MCEP, we keep the state normalization proposed (Fujimoto & Gu 2021 but other algorithms do not use it. For EQL and DQL, we use their official implementation and DQL-MCEP also built upon the released codebase Wang al.
2023
The baseline methods (TD3BC, AWAC and DQL) use the prperpparamcter recommended by their papers. TD3BC uses a = 2.5 for its Q value normalizer, AWAC uses 1.0 for the advantage value normalizer and DQL uses a = 1.0. In TD3BC-MCEP, the target policy uses ã = 2.5 and the MCEP uses af = 10. Tn AWAC-MCEP, the target policy has = 1.0 and the MCEP has x = 0.6. In DQL-MCEP, = 1.0 for target policy and a 2.5 for evaluation policy. The full list of aaper-paramcters used in the experiments
can be found in Table 2
BC
IQL
AWAC
AWAC-MCEP
TD3BC
TD3BC-MCEP
actor LR.
Le-3
3e-4
36-5
3e-5
3e-4
3e-4
actor" e LR
36-5
3e-4
critic LR
3e-4
V LR
3e-4
actor/critic network
(256, 256)
discount factor
0.99
soft. update T
0.005
dropout
0.1
Policy
Tanmaormal
Deterministic
MuJoCo Locomotion
for IQL X/X
0.7
1/X= = 3
1.0
X
0.6
a/a
2.5
a²
10.0
Table 2: The
for MuJoCo locomotion tasks.
Task
# of trajectories
# of samples
Mean of Returns
Humanoid Medium
2488
1M
1972.8
Humanoid Medium Replay
3008
0.502M
830.2
Humanoid Medium Expert
3357
1.99M
2920.5
Table 3: Dataset statistics for humanoid offline data.
A.2 Data collection and
tunning for humanoid tasks
Hyperparamcters. In this experiment, we select Top-10 Behavior cloning, TD3BC and IQL as our basc- lines. For Top-10 Behavior cloning, only 10% data of highest returns are selected for learning. For TD3BC, we searched the hyperparameter a = (0.1, 0.5, 1.0, 2.0, 3.0, 4.0, 5.0]. For IQL, we searched the expectile hy- perparameter 7 (0.6, 0.7, 0,,.0.8.9.. and the policy extraction hyperparameter À = {0.1, 1.0, 3.0). For CRR, we tune the advantage coefficiency 3 {(0.1, 0.6, 0.8. 1.0, 1.2, 5.0). For TD3BC-MCEP, we searched the ā (0.1, 0.5, 1.0, 2.0, 3.0) and E (3.0, 4.0, 5.0, 10.0). The final selected hyperparameterrs are listed in
15

---

Published in Transactions on Machine Learning Research (05/2024)
This version is con- sidered as it outperforms other baselines in (Wang el al. 2020) in complex environments such as humanoid. We also applied Critic Weighted Policy as well as an argmax version of it (CRR-argmax). These design options result in CRR, CRR-CWP and CRR-Argmax variants. In Figure D we report the most performant CRR variant. for each lask. Among all its variants, CRR-Argmax shows better performance in both the medium and the medium-replay while CRR performs the best in the medium- expert task.
Table 4
For CRR, we implement the CRR exp version based on (Hoffman et al. 2020
Humanoide Data Collection. In the Table
5
provide statistics of the collected data.
BC
IQL
TD3BC
TD3BC-MCEP
actor LR
le-3
3e-4
3e-4
3e-4
actor e LR
3e-4
critic LR
3e-4
V LR
3e-4
actor actor/critic /criti network
(256, 256)
discount factor
0.99
sofl update т
0.005
dropout
0.1
Policy
TanhNormal
Deterministic
Humanoid medinmd--
T for IQL
0.6
X/2
a/a
0.5
a²
Humanoid medium modium-replay--2
for IQL
0.6
X/ā
0.1
a/ā
0.5
1.0
a²
10
for IQL
0.6
X/X
0.1
a/a
0.5
a²
Table 4: The
for Humanoid task.
A.3 The full results for the robotics manipulation experimetns
We consider similar hyperparameter for all tasks. To obtain a fair comparison, we extensively search the hyperparameters for all baselines. The CRR is tuned among À [0.4, 0.6, 0.8, 1.0, 1.2] (1.0 selected) and uses the critic weighted policy (Wagg et al.] 2020). For TD3BC, we tune À= [1.0, 2.0, 3.0, 4.0) and select 2.0. IQL ueses 0.7, X = 3.0 as recommended in the dataset paper (Hussing et al. 2023 For TD3BC-MCEP, we use à = 2, a F = [4.0, 6.0, 8.0, 10.0] (8.0 selected).
A.4 An comparison with task-specific hyper-parameten on locomotion tasks
To investigate the task-specific optimal policy constraint strengths, we search this hyperparameter for TD3BC and TD3BC-MCEP (with â = 2.5) in each task of the locomotion set. Their optimal values and the corresponding performance improvement are visualized in Figure I1 As we observed, in 7 of the 9 tasks, the optimal policies found by TD3- outperform optimal policies found by TD3BC. In all medium tasks, though the optimal constraint strenths are the same for TD3BC and TD3BC-MCEP, TD3BC-MCEP outperformance TD3BC. This is benefitted by that relaxing the constraint of evaluation policy does not influence the value estimate. However, for TD3BC, milder constraint might cause unstable value estimate
16

---

Published in Transactions on Machine Learning Research (05/2024)
Dataset
BC
CRR
IQL
TD3BC
-MCEP
Medium
Box-PickPlace
10.8(1.3)
92.8(0.5)
93.8(2.7)
89.8(2.9)
100(0)
Box-Push
74.6(1.9)
39.2(4.2)
91.8(1.1)
93.8(1.6)
99.8(0.2)
Box-Shelf
91.8(1.2)
91.6(0.7))
98.6(0.9)
93.2(2.8)
99.2(0.7)
Tox-Trashean
8.6(2.3)
24(2.9)
0(0)
1.2(1.1)
0(0)
Dumbbell-PPcckt
38.6(2.7)
62.2(1.2)
86.8(2.0)
63.2(3.0)
70.4
Dumbbell-Puss
55.2(3.7)
20.6(1.2)
66.6(2.4)
54.0(8.6)
58.0(10.4)
Dumbbell-Shelf
40.8(1.9)
50.2(1.5)
0.6(0.4)
21.0(4.3)
44.6(11.2)
Dumbbell- Trashhan
5.2(0.7)
62(18)
87.1(3.8)
28.0(10.2)
68.2(16.1)
Hollowbox- PickPlace
42.4(3.5)
85.8(2.0)
95.2(2.4)
82.6(11.1)
92.2(3.6)
Hollowbox-PPe
0(0)
55.2(4.8)
69.4(7.5)
49.2(4.8)
98.2(110)
Hollowbox-Shadd
72.2(1.8)
94(0.7)
98.2(1.4)
95.4(1.7)
98.4(0.6)
Hallowbox-Trasiiie
0(0)
29.2(2.1)
0(0)
0(0)
0(0)
Plate-ickPPlace
0(0.2)
60.8(3.4)
1(0.3)
2.2(0.4)
0.4(0.2)
Plate-Push
0(0)
12.6(1.1)
0(0)
0(0)
25.0(11..)
Plate-Shelf
24.2(7.44
67(2.0)
99(0.4)
60.4(19.6)
99.8(0.2)
Plate Trashcan
0.2(0.2)
74(1.1)
0.4(0.2)
0.8(0.2)
0(0)
Average
29.0
56.9
55.5
45.9
59.6
Medium- Hediun-Replay
Rox-PickPlaae
0(0)
41.4(2.9)
50.8(11.5)
23.0(12.1)
0(0)
Box-Push
0(0)
2.8(0.8)
0(0)
60.8(5.1)
41.6(10.3)
Box-Shelf
0(0)
3.0(1.4)
16.6(4.6)
6.4(2.1)
49.8(14.6)
Box- Box-Trashean
0(0)
1.4(0.5)
92.3(2.6)
0(0)
76.2(6.9)
Dumbbell-PickPie
0(0)
25.6(4.6)
31.1(2.7)
8.2(6.9)
0(0)
Dumbbell-Push
4.1(2.1)
1.4(0)
3.2(1.6)
21.2(5.0)
55.8(6.0)
Dumbbell-Shell
9.8(2.3)
2.6(0.8)
11.6(6.1)
25.4(7.8)
12.0(10.1)
Dumbbell- Trashean
5.0(2.3)
17.0(4.3)
65.0(10.2)
29.0(5.6)
95.2(005)
HhllowoxxPiccPlllan
0(0)
5.6(3.3)
0(0)
6.6(5.9)
0.4(0.4)
Hollow Push
0(0)
18.2(3.0)
30.0(3.7)
23.0(12.2)
3.0(2.7)
Hollowbox-Shelf
0(0)
1.8(0.9)
61.4(4.7)
32.0(6.5)
58.4(12.6)
Hollowbox-Traslle
0(0)
4.4(0.8)
4.8(4.6)
0(0)
O(0)
Plate-PickPlacee
0(0)
31.4(2.2)
29.4(17.8)
2.2(2.0)
0(0)
Plate-Pust
0(0)
0(0)
0(0)
O(0)
10.6(9.5)
Plate-Shelf
0(0)
0.8(0.4)
0(0)
0(0)
19.4(17.4)
Plate- Trashcan
0.8(0.4)
32.8(1.1)
1.0(0.3)
0.4(0.2)
0(0)
Average
1.2
11.9
25.0
15.0
26.4
Table 5: Win rates with standard errors for Robotic Manipulation tasks. All results averaged among 5 random seeds. The win rates of the MCEP that outperform the base algorithm are bolded.
during training. In all medium-replay lasks, we found optimal constraints for TD3BC-MCEP are milder than TD3BC, which verifies the requirements of milder constraints 5.2
A.5
An investigation of other methods for inference-time action selection
The MCEP aims improve the inference time performance without incrcasing the Bellman estimato error. Previous works also propose to use the on-the-fly inference-time action selection methods. For exam- ple, @Wang et al. 2020 proposes the Critic Weighted Policy (CWP), where the critic is used to construct categorical distribution for inference-time action selection. Another simple method is selecting the action of the largest Q values, namely Argmax. In this section, we compare the performance of TD3BC and TD3BC-MCEP under different test-time action selection methods in the hidh-diimensional Humanoid tasks.
The results are presented in Table and 7 Both the Argmax and the CWP methods select an action from an action set. We generate this action set by adding Gaussian noise to the outputs of the deterministic policy. The sid is the noise scale and N is the size of this action set. From the results, observe that
17

---

Published in Transactions on Machine Learning Research (05/2024)
Optimal Constraint Strengths
Performance Difference
10
Percent Difference
0.1
exprasion
0.0
of to3be moeg. with 2.5
-0.1
ha-me:
warme
ha-m
wa-th
haette ho-me wa-me heeth ho-m wirm havrr houm we-m!
Figure 11: Left: Optimal E values for the evaluation policy of TD3BC-MCEP, with a fixed 2.5 for the target policy. Optimal values for TD3BC. Red areas indicate the a values for TD3BC that raise Q-value explosion (in one more training of 5-seed training). Right: Performance difference (with standard errors) between the evaluation policy of TD3BC-MCEP and the actor of TD3BC, using the a (a) values shown in the left figure.
CWP and Argmax help improve the performance of both the TD3BC and TD3BC-MCEP, It is worth noting that, in medium task, the Argmax method improves the TD3BC to the same level as TD3BC-MCEP But in meidum-replay and medium-erpert tasks, the improved performances are still worse than the original TD3BC-MCEP (without using action selection). On TD3BC-MCEP, applying Argmax and CWP further improves policy performances.
In conclusion, the inference-time performance could be improved by utilizing the inference-time action se- lection methods but MCEP shows more significant policy improvement and does not show conflict with these action selection methods.
Table 6: TD3BC with inference-time action selection. The original policy has returns 2483.9, 965.4 and 3898.2 2 for medium, medium-replay and medium-expert, respectively. Standard errors are also reported.
Argmax
CWP
Game
Nistd
0.01
0.02
0.05
0.02
0.05
20
2462.4(1886
2044.5//71.3)
3098, 9(149.2)
3511.1(264.7
2441.4(217-4
2689 8(323.3
2303.4
(11332(411.5)
medium
2564.2(180.1)
2836 (336.00399.7)
6(
3156.3/272.4)
2159.8((39.8)
2588. 6(181.)
2402. 1(180.5))
2839.6(165.8)
100
2857.0(96.7
2309 ((29.)
3122 (12220)222.7)
3200.8(314.4)
2007. 6007.6(140.9)
2685. 8(335.
2722 0022255305.00
2684.1
20
895.3(44.1)
1642.30 (1.30133333
1136 1136.7(135.0 7(135.8)
1524.1(125.8)
973 9778977777)
931 966 10)
932 2(42.4)
1242.7(116.5)
medium-replay
50
891.))0033
976.7(18.9)
1160. 2(72.8)
1664.9(248.
971.7
1030.3/90
1002 1(67.7)
1171.1(124.))
971.7 (69.7))0.5)
1649.0(35.6)
1292 2(144.2)
1874.7(179.8)
874.20 224.2(41.00
1023 5/85.0
173 0.740)748)
1232-9(1172)
7(345.8))
4068 ((7.7)
4131 (((99-7))
4606.3()06-6)
4181.1 1166.1
4478.3(17433
3004.0(166.1
(6367(263341
medium-coport
50
4460. 8/300)
4012 4012.2(318.7)
4612.9(127.7)
4603.0 (137.5)
3987. 03877.0288.11
4069 9/ 206.0)
3986. (323.66
4214.7(157.8)
4130-8(301.2))
4141.7((28.4)
4158-3(30288)
4421.4(130.4)
4145.3(26220)
3634.663380. 4)
3933.9(240.4)
3788.3(246.1)
Table 7: TD3BC-MCEP with inference-tinn time action selection. The original policy has returns 2962.8, 4115.6 and 4829.2 for medium, medium-replay and medium-expert, respectively. Standard errors also reported.
Argmax
CWP
Game
N)edd
0.01
0.05
0.1 0.01
0.02
0.05
20
2368.2(221.0 2)
2871 4(334.)))
2914.. 1.8)
33912 8(310.)
2670-5(262.8)
2710.0(96.8)
3146.5(216.6)
2087. (2r.9((44.0)
mediamm
2822.1(164.0
30H6.3(287.22)
3283.9
3861.7(225.2)
264129(112..)
2787.1(164.9)
2718.9(344.9)
2841.7(248.7
100
3405.1(326. 9)
2806. 2805.2(191.8)
3261.5(310.4)
3751.3/ 190.7)
3002.3(312.1)
2896.8(192.2
(163.)(163.1)
2727.2 2727.2(281.9)
4277.3(708
4071.5(70224)
4092.7(694.1)
4253.4(6330 ((6.34))3777)
((08988)
4200.0(652.1)
4700.4
4167 00..0662.11
modium-replay
50
4225.566991.. 8)
4169-7(681.4)
4028.4 46234
4210.8(7100 8(7102)
(161-)
42191
4876.7(760.6)
4230336629
100
1190.3(891.0) d)
3966.4(639.99 9)
11.38 (694.))
1270 (695.00
4328.5/ ((78.5(733.3)
1266.9 1266.9(638.8)
1275.7(671. 1.3)
4142.5(3968 7)
20
4752.8(202.
4066.7:92.6)
4880.3/
4887. ((2881
4736-4(266.5)
4710.8(226.7)
4748.7(177) (1.771177.55
4942.1
mediumm-sppert
50
(167 6)
501 8.2(73.6)
4614.2 (15100)))0000
4800 0/900.91100 158.7)
5053.4/77 4/73.4
5001 (85.00)
4808 8(183.4)
4670.6(204.2
4616.8(217.9)
4800:9(145-4)
4700.9(179.7)
0(8.0))8885)
(1257.3)
4770.6(86.3)
(931.3(176.6)
4865.3(146.9)
18

---

Published in Transactions on Machine Learning Research (05/2024)
A.6 The design option of how the evaluation policy update
As the evaluation policy is not involved in the actor-critic's iterative update, might want to update the evaluation policy after the actor-critic converges, namely afterward updates. While this is valid design option, our method simultaneously updates the target policy and the evaluation policy (simultaneous updates). In this manner, their updates can be parallelized and no further time is required based on the actor-critic training. This parallelization can significantly reduce the training time for methods of slow policy update (c.g. DQL). Figure 12 and 13 present the convergence for these two design options. From the results, we observe faster convergence of afterward updates in some lasks. However, there are also many lasks where the afterward updates method converges after million steps.
0.0
62
0.8
0.6
0.6
1.0
0.0
0.2
0.0
0.3
0.2
0.4
0.6
0.8
1.0
120
130
100
000
20
0.6
1.0
0.0
0.2
0.2
0.4
0.6
0.8
120
120
120
100
100
100
80-
41
60
90
40
atlerward uptiales
20
20-
dimuitnncuus updates
0.6
10
medium
medium-replay
268
medium-expert
Figure 12: Episode returns with standard errors of simultaneous updates and afterward updates for the evaluation for TD3BC-MCEP. First row: halfcheelah. Second row hopper. Third row: walker2d.
100
0.0
0.2
0.4
0.0
0.8
1.0
0.0
13.2
0.4
08
0.5
0.0
0.2
0.4
0.0
0.
10
220
120
130
100
100
1.0
80
50
60-
14
40
40
20
20
20
0.4
0.5
02
0.4
0.S. on
1.0
medium
165
medium-replay
Ted
medium-expert
165
Figure 13: Episode returns with standard errors of simultaneous updates and afterward updotes for the evaluation for AWAC-MCEP. First row: halfcheetah. Second row hopper. Third row: walker2d.
19

---

Published in Transactions on Machine Learning Research (05/2024)
A.7
The full results for estimated Q values of the learned evaluation policies
Figure 14 and Figure 15 show the visualization of the estimated Q values achieved by the target policy and evaluation policy.
target policy evaluction policy
(a) medium-expert (b) medium
(c) medium-replay
(d) random
Figure 14: TD3BC-MCEP. First row: halfcheelah. Second hopper. Third rows walker2d.
03
avollation policy
(a) medium-eett exppet
(b) medium
(c) medium-replay
(d) random
Figure 15: AWAC-MCEP, First row: halfcheetah. Second row: hopper. Third row: walker2d.
20


Reviews:
[{"review_id": "zGbxQBpmjc", "reviewer": "Reviewer_5tm2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "== Strengths ==\n\nThis paper is generally well-written, and the logical flow is clear. I would say this paper is also well-motivated and proposes an interesting test-time inference algorithm. The resulting method is very simple, and the authors provide some figures and a toy example to show the readers the key idea behind their method, which I personally like very much. It is also good to see that the authors enrich the introduction part and make the motivation of this paper clearer.\n\nThe authors combine their method with three off-the-shelf offline RL algorithms, and conduct some experiments on the D4RL locomotion datasets. The authors also conduct experiments on the Humanoid datasets, where the authors collect the corresponding static datasets by themselves. Furthermore, the authors conduct experiments on Robotic Manipulation tasks. One can observe performance improvement by building MCEP upon numerous base algorithms. To summarize, the strengths and the advantages of this manuscript are\n\n- this paper is well-written with a clear logic flow\n\n- the core idea and the resulting method of this paper is quite simple and easy to implement\n\n- the improvements from the proposed method are significant on many base algorithms\n\n- the authors provide source codes, and I believe that the results presented in this paper are reproducible\n\n- the authors consider many tasks and extensively examine the performance of their proposed method\n\n- the limitation part is well-stated and highly appreciated\n\nOverall, I believe this paper is qualified to be published in TMLR\n\n== Weaknesses ==\n\nSome of my concerns in ICLR2024 review are addressed in the current version. My only criticism to this version is the lack of statistical significance, e.g., error bar and standard deviation. **This applies to almost all of the tables and figures presented in the paper**.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I am honored to review this paper again and I am Reviewer VtJK of this paper at ICLR 2024. I summarize the contributions of this paper below:\n\nIn policy constraint offline reinforcement learning (RL) algorithms, it is a common practice that the constraints for both value learning and test time inference are the same. This paper argues that such a paradigm may hinder the performance of the agent during test time inference. To address this issue, they propose the Mildly Constrained Evaluation Policy (MCEP) for test time inference. The idea is quite simple and the implementation is also easy. MCEP has the same objective function as the policy trained during the offline phase, but it does not participate in the policy evaluation phase. The authors show that by doing so, the performance of the offline RL agents can be improved.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please add error bars in Figure 7, Figure 8, Figure 11\n\n- Please also report standard deviations in Table 5, Table 6, Table 7", "broader_impact_concerns": "I believe no broader impact concerns are needed to be specified.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4Jtk0rWcQf", "reviewer": "Reviewer_upJw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "My main concern is that I don't understand how MCEP training is different from the standard actor-critic method. In particular, why line 8 of Algorithm 1 is useful? For instance, the information in line 8 is not leveraged since the computation in Line 9 or Line 6 does not use any information related to psi. The algorithm will run seamlessly without Line 8. In this sense, I don't understand where the improvement from Table 1 comes from. \n\nSecond, is there a principled way to compute the hyperparameter lambda so the algorithm can be better than the standard actor-critic? In addition, how's other hyperparameter tuned?\n\nLastly, it would be nicer if the study could provide some sample complexity guarantees. \n\nSmall questions:\n\n1. In Algorithm 1, it should be gradient rather than loss itself in the update;\n2. The one-step method [1] seems related, the author should include it and discuss the pros and cons (compared to the current work).\n[1] Offline Reinforcement Learning with Closed-Form Policy Improvement Operators, ICML23.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the policy constraint methods from a novel aspect by showing that the learned value function is not well exploited under the tradeoff between value learning and policy performance. To handling this, it proposes use an extra mildly constrained evaluation policy with a more constrained target policy, to achieve better policy performance and stable value learning simultaneously.\n\nThe empirical evaluation on D4RL MuJoCo locomotion, high-dimentional humanoid and a set of 16 robotic manipulation tasks show that the MCEP obtains significant performance improvement for policy constraints methods. Related ablation study is also conducted to for the study.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please answer my questions in above.", "broader_impact_concerns": "This is a study that does not raise Broader Impact Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "M5p0c7mR7p", "reviewer": "Reviewer_Wivj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The approach is simple, yet general and effective. \n* Results demonstrate that it surpasses various strong baselines in the tasks it was tested on. \n* The paper is mostly well-written, and I appreciate how the authors carefully explain their motivation, supporting it with empirical evidence.\n\nWeaknesses:\n\n* The approach requires increased effort in hyper-parameter tuning, which could be challenging when deploying in real-world scenarios where simulators are unavailable. \n* The introduction is somewhat obscure and fails to effectively convey the significance of the contributions. \n* Some of the plots and their captions are difficult to understand. \n* Additionally, several parts of the paper could be reorganized to improve its overall flow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an offline reinforcement learning (RL) approach that separates value learning from policy evaluation. This separation enables the learning of a better policy in the latter stage without affecting the stability of the former stage. To motivate their approach, the authors empirically demonstrate that the critic learned from standard offline RL suggests a stronger policy than the one learned directly. Results on an offline RL benchmark, D4RL, showcase the effectiveness of this approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The second and third paragraphs require revision for clarity. Currently, there are two main issues:\n  *  The language used in these paragraphs doesn't effectively connect with the experimental questions in section 5. It's initially unclear why those questions are posed. To improve this, try to avoid alternating between terms like \"critic\" and \"value function\", especially since they refer to the same concept in this paper's context. Consistently using one term could make the connection more apparent. \n  * The necessity of training stability, despite good evaluation performance, isn't immediately evident. If training instability poses a safety risk or if steady improvement is crucial for optimal final performance, this should be explicitly stated. The discussion in the experiment section, which suggests the critic implies a better policy than the learned one, offers a compelling motivation. This point could be introduced earlier to engage readers sooner.\n  * Consider integrating the last four sentences of the second paragraph into the third paragraph, where your method is introduced. These sentences, which currently seem to pertain only to evaluation, actually provide valuable insights into the approach's details and benefits. Their current placement might imply they are relevant only in the evaluation context, while the discussion of the proposed approach itself is somewhat brief.\n\n- Introducing a reference to Figure 2 earlier in the paper, possibly in the introduction or just after section 3, could help readers better grasp the issue your research addresses.\n\n- Algorithm 1 needs more precision. Specify whether each update computes loss on the entire dataset or merely a batch of data. Clarify if all models are updated in each iteration with the same batch and if a batch parameter should be included in all the loss functions. Additionally, confirm if all learning rates are identical.\n\n- Please clarify how training and evaluation conditions differ. Are the tests conducted in the training environments, or are the testing environments sampled from the same distribution as the training environments?\n\n- Figure 4 is confusing due to its non-informative caption. Clarify whether these results pertain to training or testing environments. The term \"divergence\" in value estimate needs definition: does it refer to a high variance in the Q function or a divergence in the curves of different hyper-parameter values? If the variance indicates instability, consider showcasing only the variance over time. Clearly indicate upfront what signs of instability (e.g., larger variance) readers should look for. Additionally, it's unclear why the evaluation performance isn't plotted, especially since it's mentioned in the text. Clarify whether the second row shows training or evaluation performance. The term \"evaluation performance\" itself is somewhat vague; specify whether it refers to performance in test or training environments.\n\n- In all tables and figures where confidence intervals are reported or plotted, please specify the type of intervals used (e.g., standard error, X% CI derived from standard error, etc.).", "broader_impact_concerns": "Authors can potentially discuss the cost of tuning the hyper-parameters and how it effects the safety of this method in real-world applications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have engaged with the reviewers and made changes to the paper based on their suggestions. The review discussion pointed out that the authors can further improve the manuscript and have the following suggestion:
*  The technical motivation could be written more clearly and some of the terminologies can be used with greater consistency.

I'm recommending acceptance, but strongly encourage the authors to consider making the above minor writing improvements for camera ready.
Decision: Accept (poster)