A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning
Authors: Benjamin Eysenbach
Matthieu Geist
Sergey Levine
Ruslan Salakhutdinov
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
regularization
one-step RL
theory

Abstract: As with any machine learning problem with limited data, effective offline RL algorithms require careful regularization to avoid overfitting. One-step methods perform regularization by doing just a single step of policy improvement, while critic regularization methods do many steps of policy improvement with a regularized objective. These methods appear distinct. One-step methods, such as advantage-weighted regression and conditional behavioral cloning, truncate policy iteration after just one step. This ``early stopping'' makes one-step RL simple and stable, but can limit its asymptotic performance. Critic regularization typically requires more compute but has appealing lower-bound guarantees. In this paper, we draw a close connection between these methods: applying a multi-step critic regularization method with a regularization coefficient of 1 yields the same policy as one-step RL. While practical implementations violate our assumptions and critic regularization is typically applied with smaller regularization coefficients, our experiments nevertheless show that our analysis makes accurate, testable predictions about practical offline RL methods (CQL and one-step RL) with commonly-used hyperparameters. Our results  that every problem can be solved with a single step of policy improvement, but rather that one-step RL might be competitive with critic regularization on RL problems that demand strong regularization.

A CONNECTION BETWEEN ONE-STEP RL AND CRITIC REGULARIZATION IN REINFORCEMENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
As with any machine learning problem with limited data, effective offline RL algo- rithms require careful regularization to avoid overfitting. One-step methods perform regularization by doing just a single step of policy improvement, while critic regular- ization methods do many steps of policy improvement with regularized objective. These methods appear distinct. One-step methods, such as advantage-weigee gression and conditional behavioral cloning, truncate policy iteration after just one step. This "carly stopping" makes one-step RL simple and stable, but can limit its asymptotic performance. Critic regularization typically requires more compute but has appealing lower- bound guarantees, In this paper, we draw a close connection be- tween these methods: applying a multi-step critic regularization method with a reg- ularization coefficient of 1 yields the same policy one-step RL. While our theoret- ical results require assumptions (e.g., deterministic dynamics), our experiments nev- ertheless show that our analysis makes accurate, testable predictions about practical offline RL methods (CQL and one-step RL) with commonly used hyperparameters.
re-
INTRODUCTION
Reinforcement learning (RL) algorithms tend to perform better when regularized, especially when given access only limited data, and es- pecially in batch (i.e., offline) settings where the agent is unable to collect new experience. While RL. algorithms can be regularized using the same tools as in supervised learning (e.g., weight decay, dropout), we will use "regulariza- tion" to refer to regularization methods unique to the RL setting. Such regularization methods include policy regularization (penalizing the pol- icy for sampling disstribbttiinoo action) and value regularization (penalizing the critic for making large predictions). Research on these sorts of regularization has grown significantly in recent years, yet theoretical work studying the tradeoffs between regularization methods remains limited.
critic regularization
I-sste (a
B(a
(a
n-step RL
more regularization
less regularization
Figure I: Both n-step RL and critic regularization can interpolate between behavioral cloning (left) and un- regularized RL (right) by varying the regularization pa- rameter. Endpoints these regularization paths are the same. We prove that these methods also obtain the same policy for an intermediate degree of regularization.
Many RL methods perform regularization, and can can be classified by whether they perform one or many steps of policy improvement. One-step RL methods (Brandfonbrener et al. |02 Peng et al. 2019 Peters & Schaal 2007| Peters et al. 2010) perform one step of policy iteration, updating the policy to choose actions the are best according to the Q-function of the behavioral policy. The policy is often regularized to not deviate far from the behavioral policy. In theory, policy iteration
can take large number of iterations (Ó(SS) ()) (Scherrer 2013)) ) to converge, so one-step RL (one step of policy iteration) fails to find the optimal policy on most tasks. Empirically, policy iteration often converges in smaller number of iterations (Sutton & Barto 2018] Sec. 4.3), and the policy after just a single iteration can sometimes achieve performance comparable to multi-step
RL methods (Brandfonbreener et al.) 2021). Critic regularization methods modify the training of the value function such that it predicts smaller returns for unseen actions (Kumar et al. 2020; Chebotar
et al., [222 Yu et al. 2021 Hatch et al. [022 Nachum et al.) 2019 An et al. [202 Bai et al. 2022] Buckman et al., 2020). Errors in the critic might cause it to overestimate the value of some unseen actions, but that overestimation be combated by decreasing the values predicted for all unseen

---

actions. In this paper, we will use "critic regularization" to specifically refer to multi-step methods
that use critic regularization.
In this paper, we show that certain type of actor and critic regularization can be equivalent, under some assumptions (see Fig. D. The key idea is that, when using a certain TD loss, the regularized critic updates converge not to the true Q-values, but rather the Q-values multiplied by importance weight. For the critic, these importance weights mean that the Q-values end up estimating the expected returns of the behavioral policy (QP, in many one-step methods (Peters et al. 2010 Peters & Schaal 2007, Peng et al. 2019, Brandfonbrener et al., (02T)), rather than the expected returns of the optimal policy (Q""), ) For the actor, these importance weights mean that the logarithm of the Q-values includes a term that looks like KL divergence. So, optimizing the policy with these
Q-values results in a standard form of actor regularization.
The main contributions of this paper are as follows:
We prove that one-step RL produces the same policy as a multi-step critic regularization method, for certain regularization coefficient and when applied in deterministic settings.
We show that similar connections hold for goal-conditione RL, as well as other RL settings. We provide experiments validating the theoretical results in settings where our assumptions hold. We show that the theoretical results make accurate, testable predictions for practical offline RI. methods, which can violate assumptions.
2 RELATED WORK
Regularization has been applied to RL in many different ways (Neu et al. 2017 Geist et al. 2019), and features prominantly in offline RL methods (Lange et al. 2012 Levine et al. (020). While RL. algorithms can be regularized using the same techniques in supervised learning (e.g.. weight decay, dropout), our focus will be on regularization methods unique to the RL setting. Such RL-specific regularization methods can be categorized based on whether they regularize the actor or the critic.
One-step RL methods (Brandfonbrener et al. 2021 Gülçehre et al. 2020 Peters & Schaal 2007
Peng et al. 2019); Peters et 20100 Wang al., (018)) apply single step of policy improvement to the behavioral policy. These methods first estimate the Q-values of the behavioral policy, either via regression or iterative Bellman updates. Then, these methods optimize the policy to maximize these
Q-values minus an actor regularizer. Many goal-conditioned or task-conditioned imitation learning
methods (Savinov et al. 2018 Ding et al. al. 2019 [Sun et al.) 2019 | Ghosh et al. 2020 Paster et 20200
Yang et a |222 Srivastava et al. 2019 Kumar et al. 2019 Chen et al. 202 | & Sermanet
2021 Li et al. 2020 Eysenbach et al.
2020a) also fits into this mold (Eysenbach et al.) 2022],
yielding policies that maximize the Q-values of the behavioral policy while avoiding unseen actions.
Note that non-conditional imitation learning methods do not perform policy improvement, and do not fit into this mold. One-step methods are typically simple implement and computationally efficient.
Critic regularization methods instead modify the objective for the Q-function so that it predicts lower returns for unseen actions (Kumar al. 2020 Chebotar et al. [202 Yu et al. 2021 Hatch et al.)
2022, Nachum et al., [2019] An et al.) [202 | Bai et al. [022 Buckman et al.) 2020). Critic regular- ization methods are typically more challenging
implement correctly and more computationally
demanding (Kumar et al.) 2020; Nachum et al. 2019 Bai et al.
2022 An et al. 2021). but can lead
to better results on some challenging problems (Kostrikov et al. (021) Our analysis will show that one-step RL is equivalent to a certain type of critic regularization.
3
PRELIMINARIES
We start by defining the single-task RL problem, and then introduce prototypical examples of one-step RL and critic regularization. We then introduce an actor critic algorithm we will use for our analysis.
3.1 NOTATION
We assume an MDP with states s, actions a, initial state distribution (o())). dynamics p(s' (ss a), and reward function r(s, a) We assume episodes always have infinite length (i.e., there are no terminal states). Without loss of generality, we assume rewards are positive, adding a positive constant to all
If the reward also depends on the next state, then define r(s, a) Epc( (s,,,,
2

---

rewards make them all positive without changing the optimal policy. We will learn Markovian policy (a 8) maximize the expected discounted sum of rewards:
8
max E m((r)
(r((s,, ~ po(so)
where x(T) = p(so) Io (aa se)p(st41
St, at)aa) is the probability of policy и sampling an We define Q-values for policy T(a (s) as
infnite-length trajectory (so: a0, ap."..).
Q" Bx(r)
Σ arrss so 5, 00
Note that the reward being positive implies that the Q-values are also positive, Q* a) > 0. Since we focus on the offline setting, we will consider two policies: B(a s) is the behavioral policy that collected the dataset, and 5( s) is the online policy output by the algorithm that attempts to maximize the rewards. We will use p(8, a, 8') to denote the empirical distribution of transitions an offline dataset, and p(s, a.) and p(s) denote the corresponding marginal distributions. The behavioral policy is defined as ((a s) p(a| s).
3.2 EXAMPLES OF REGULARIZATION IN RL.
While actor and critic regularization methods can be implemented in many ways, we introduce two prototypical examples below to make our discussion more concrete.
Example of one-step RL: Brandfonbrener et al. 22021 One-step RL first estimates the Q-values of the behavioral policy (Q"(s, a)). and then optimizes the policy to maximize the Q-values minus actor regularizer. While the actor regularizer can take different forms and the Q-values can be learned via regression, we will use reverse KL regularizer and TD-style critic update so that the objective is similar to critic regularization:
max Epca)m(ala) [Q'(s, (s, a) + A(log B(a | (s) -) log log {)}} where Q (0(8,)) = lim Qe(s,a)
(1)
and Qun + arg min Bp(c,a)
and ( & r(s, 1 peeel -
[Qe(s d')]
Q
A(u 9(11k/))
where X is the regularization coefficient and 3(a 8) is an estimate of the behavioral policy, typically learned via behavioral cloning. Like most TD methods (Haarnoja al. 2018 Mnih et al. 2013
Fujimoto et al.] 2018). the TD targets y are not considered learnable. In practice, most methods do not solve optimize the critic to convergence each step, instead taking just few gradient steps before updating the TD targets. This one-step critic loss is different from the multi-step critic losses used in other RL methods (e.g., TD3, SVG(O)) because uses the TD target 20 (s, a) (corresponds to fixed policy) rather than y (s, a) (corresponding
sequence of learned policies). One-step RL amounts to performing one step of policy iteration, rather than full policy optimization. While truncating the iterations of policy iteration can be suboptimal, it can also be interpreted a form of early stopping regularization.
Example of critic regularization. Kumar et al. (2020). CQL (Kumar et al., 2020 modifies the standard Bellman loss include an additional term that decreases the values predicted for unseen actions. The actor objective maximize Q values; some CQL implementations also regularize the actor loss (Hoffman et al. 2020 Kumar et al. 2020 The objectives can then be written as
max Ep(o)m(ate) [Q" (s, a)), where Q" (s lim Q,(s,a)
- y (s, ( ) + (Eptointet) [Q(s,)) - Epte)s(aia) [Q(s.))))
and Q++1 arg min Ep(s.a)
The second term decreases the Q-values for unseen actions (those sampled from <(a s)) while the third term increases the values predicted for seen actions (those sampled from the behavioral policy 3(a s)). Unlike standard temporal difference methods, the CQL updates resemble competitive game between the actor and the critic. In practice, this cyclic dependency can create unstable learning (Kumar et al. 2020; Hoffman et al. 2020
3

---

3.3 HOW ARE THESE METHODS CONNECTED?
Prior work has observed that one-step methods and critic regularization methods perform similarly on many (Fujimoto & Gu 2021 Emmons et al. 202T) (but not all (Kostrikov et al.) (02T)) tasks. Despite the differences in objectives and implementations of these two methods (and, more broadly, the actor/critic regularization methods for which they are prototypes), are there deeper, unifying connections between the methods?
In the next section, we introduce a different actor-critic method that will allow us to draw a connection between one-step RL and critic regularization. We experimentally validate this equivalence in Sec. 5.1 Despite its difference from practically- used methods, such as one-step RL and CQL, we will show that it makes accurate predictions about the behavior of these practical methods (Sec. 5.2 and 5.3
3.4 CLASSIFIER ACTOR CRITIC
To support our analysis, we will introduce a new actor-critic algorithm. This algorithm is similar to prior work, but trains the critic using cross entropy loss instead of an MSE loss. We introduce this algorithm not because we expect it to perform better than existing actor-critic methods, but rather because allows to make precise a connection between actor and critic regularization. This method treats the value function like a classifier, so we will call it classifier actor critic. We will then introduce regurrregulrr and retiicrrgullrrized versions of this method. The subsequent section (Sec. D will show that these two regularized methods learn the same policy.
The key to our analysis will be to treat Q-values like probabilities, so define the critic loss in terms of cross-entropy loss, similar to prior work (Kalashnikov et al. [2018] [Eysenbach et al. 202T). Recalling that Q-values are positive (See. (.T], we transtorm the Q-values to have the correct range by
using Q+1 E
[0, 1). We will minimize the cross-entropy loss applied to the transformed Q-values:
Eptni)) |CE ((s,a) a) Qr (8.00)
y VT. (s.
(3)
-Bplona) y TT (s,a)
log Q(s,a)
+( + (=, +1 log Q(s, a) +1
( +1
const. -Bple,a)
y (x, log Q(s,a) +1 log Q(5,0) +
((s,a)
a Lenai (2,7x.0)).
(4)
In the last line we scale both the positive and negative term by Q a) + 1. a choice that does not change the optimal classifier but reduces notational clutter. When the TD target can be computed exactly, solving this optimization problem results in performing one SARSA update: ((s,a) f r(s, a) + (s'',, ('''a') a') (see Lemma 4. 1. Thus, by solving this optimization problem many times, each time using the previous Q-value to compute the TD targets, we will converge to the correct Q-values (see Lemma 4.1 The actor objective is to maximize the expected log of the Q-values:
max Luctor TT) 7
(p()))()))) [log( (sT((((ss))) ())a))), where (s, lim Qs(s,a)
and Q++1 = arg min Cenne Q2 y (.))
Q
While most actor-critic methods do not use the logarithm transformation, prior work on conditional
behavioral cloning (e.g., (Savinov et al., 2018 Ding et al., 2019, Sun et al. 2019 Ghosh et al. 2020; SSriastava et al. 2019)) implicitly includes this transformation (Eysenbach et al. [022]. p. In the absence of additional regularization, the optimal policy T(a) 8) T(a arg max, Q(+ a') ) the same as the optimal policy for the standard actor objective (without the logarithm). We next introduce one-step version of this method, well critic regularization variant that resembles CQL. While we will implicitly use a regularization coefficient of 1 below, Appendix [B. discusses versions of classifier actor critic with varying degrees of regularization.
One-step RL. To make classifier actor critic resemble one-step RI. (Brandfonbrener et al. 202 1) we make two changes: estimating the value of the behavioral policy and adding a regularization term to the actor objective. To estimate the value of the behavioral policy, we modify the critic loss to sample the next action a' from the behavioral policy (i.e., use 9000 a) rather than n.Q (s)
4

---

We also regularize the policy by adding relative entropy term the actor loss, analogous to the reverse KL penalty used in one-step RL:
max (p(e)talle) log (s, a) + S(a) s) log (a s) where (?(s = lim Qe(s.a)
(6)
t-000
and Q41 = arg min Lerie (Q, 9:
Q
In tabular settings, this critic objective estimates the Q-values for 3(a (Appendix Lemma 4.1
Critic regularization. To emulate CQL, modify the critic loss (Eq. 4 by adding penalty term that decreases the values for unseen actions. Whereas CQL applies this penalty to the Q-values directly, we will apply it to the logarithm of the Q-values
max (p(e)acala) [lg Q (s, a)], where Q (s,
Qe(s,a)
a0+1(8, arg min (Q, + (p(a))(al)) [log(Q(s, a) + +1] Bacspca)))) [log(Q(s, +
Cate (..)
A CONNECTION BETWEEN ONE-STEP RL AND CRITIC REGULARIZATION
This section provides our main result, which that actor and critic regularization yield the same policy under settings. The key proving this connection will be analyze the Q-values learned by critic regularization. While we mainly focus on the single-task setting, Sec. S.2. deseribes how similar results also apply to other settings, including goal-conditioned RI., example-based control, and settings with smaller degrees of regularization. All proofs are in Appendix A
To relate one-step RL to critic regularization, methods. We first show that the classifier critic converges to the correct Q-values:
start by analyzing the Q-values learned by both
Lemma 4.1. Assume that states and actions are tabular (discrete and finite), that rewards are positive, and that TD targets can be computed exactly (without sampling). Incrementally update the critic by solving a sequence of optimization problems:
Q+t1 arg min Leciti (Q, yz
In the limit, this sequence of Q-functions will converge to Q": lim Qe(s.a) = (s, for all states and actions a.
Because one-step RL trains the critic using Lenie (Q, 3.Q). it learns Q-values corresponding to (s,ss)). When regularization added the critic updates, learns different Q-values. Perhaps surprisingly, this regularization means that our estimates for the value of policy T(a (s) look like the value of the original behavioral policy:
Lemma 4.2. Assume that states and actions tabular (discrete and finite), that rewards positive, and that TD targets can be computed exactly (without sampling). Incrementally update the critic by minimizing sequence of regularized critic losses using policy n and hyperparameter X 1:
Q++1 f- arg min Ci (Q. 99)).
Q
In the limit, this sequence of Q-functions will converge to the Q-values for the behavioral policy (B(a) 8)), weighted by the ratio of the behavioral and online policies:
lim Qe(s, a) (s,aa))( s) E-000
for all states s and actions a.
aTaa s)
FFrm dimensional analysis perspective Huntley 1967). this choice makes sense because it allows the penalty term to have the same "units" as the critic loss: log Q-values. A second motivation for regularizing the logarithm that the actor loss logarithm.
5

---

Proof sketch. The ratio K(as))
*[a|s) above is importance weight. Ordinarily, TD backup for policy
aa 5) would entail sampling an action ~ Taa s). However, this importance weight means that TD backup is effectively performed by sampling an action [(a s). Such a TD backup resembles the TD backup for g(a s). The full proof is in Appendix
Intuitively, this result says that critic regularization reweights the Q-values to assign higher values to in-distribution actions, where B(a s) is large. An unexpected part of this result is that the Q-values correspond to the behavioral policy. Said in other words, critic regularization added to a multi-step RL method (one using y (*.(((s, a)) yields the same critic one-step RL method (one using . (s,a)). Our main result is a direct corollary of this Lemma:
Q:
Theorem 4.3. Let behavioral policy 3(a 8) be given and let Q* a) be the corresponding value function. Let (a 8) be an arbitrary policy (lypically learned) with support constrained 3(a 8) (i.e,, (a 8) > = 3(a 8) > 0). Let Q. (s, a) be the critic obtained by the regularized critic update (Ég. 7 to this policy with À = 1. Then critic regularization results in the same policy as one-step RL.
En(a log (5, a)] = Brls [lo (* a) + log B(a - log (a 8) for all states s.
Since both forms of regularization result in the same objective for the actor, they must produce the same policy in the end. While prior work has mentioned that critic regularization implicitly regularizes the policy (Yu al. 2021). this result shows that under the assumptions stated above, the implicit regularization of critic regularization results in the exact same policy learning objective as one-step RL. This equivalence holds when 1, and not necessarily for other regularization coefficients. Appendix (a.]dxxxxBB]sss sshoss how variant of this result that includes an additional regularization mechanism does apply to different regularization coefficients. This connection between one step RL and critic regularization concerns their objective functions, not the procedures used to optimize those objective functions. Indeed, because practical offline RL algorithms sometimes use different optimization procedures (e.g.. TD vs. MC estimates of Q a)), they will incur errors in estimating Qa (s. a), violating Theorem 4.3 S assumption that these Q-values are estimated exactly.
4.1 LIMITATIONS
Our theoretical analysis makes assumptions that may not always hold in practice. For example, our results use critic loss based the cross entropy loss, while most (but not all (Kalashnikov et al.) 2018; Eysenbach et al. 2020b)) practical methods use the MSE. Our analysis assumes that critic regularization arrives at equilibriu, and ignores errors introduced by function approximation and sampling. Nonetheless, our theoretical results will make accurate predictions about practically-used offline RL methods.
4.2 EXTENSIONS OF THE ANALYSIS
We extend this analysis in three ways. First, we also show that a similar connection can be established for lesser degrees of regularization (A < 1) (see Appendix B.I). Second, we show that a similar con- nection holds for RL problems defined via success examples (Pinto& & Gupta 2016 Calandra
Tung et al. 2018 Kalashnikov et al., 2021] Singh et al. 2019 Zolna et al. 20l0nCallndra et al. 2017 Eysenbach et al.
2021) These results existing actor-critic method, rather than classifier actor critic (see Appendix Third, we extend our analysis to multi-task settings by looking at goal-conditioned RL problems.
5 NUMERICAL SIMULATIONS
Our numerical simulations study whether the theoretical connection between actor and critic regular- ization holds empirically. The first experiments (S1c will use classifier actor-critic, and we will expect the equivalence to hold exactly in this setting. We then study whether this connection still holds for practical prior methods (one-step RL and CQL). which violate our assumptions. We study these commonly-use methods in both tabular settings (Sec. (.2) and on benchmark offline RL task with continuous states and actions (Sec. (.3). We do not expect these methods to always be the same (see, e.g., Kostrikov al. (2021 Table 1)), and will focus our experiments on critic regularization

---

1004
RROWOWO 1005
edaedeed roluTe
expedded reburr
actor regubettzation
regudeitation
ector reguler reguterization
MDPs where regularization decreases returns.
MDP where regularization in-
creases returns.
Figure 2: Actor and critic regularization produce identical policies. Across three tabular settings, plot the action probahilities (a s) for the policies produced one-step RI. and critic-regularized classifier actor-critic (R² 2 0.999). also plot the action probabilities for a policy learned by an unregularized policy to confirm that the equivalence between one-step RI. and critic regularization is not coincidence.
-10
(a) reward function
(b) Q-learning
(c) One-step RL
(d) CQL( 10)
(e) CQL(A 0.1)
Figure 3: CQL behave like one-step RL. We design gridworld (a) so that one-step RL learns suboptimal policy. For the three cells highlighted in blue, the optimal policy (b) navigates towards the high- reward state (green) while one-step RL policy (c) navigates away from the high-reward state. (d) CQL with a large regularization cocfficient exhibits the same suboptimal behavior as one-step RL, taking actions that lead away from the high-reward states. (e) CQL with small regularization coefficient behaves like Q-learning. For clarity, only show the argmax action in each state; omit the when the argmax action is "do nothing".
with moderate regularization coefficients. See Appendix experiments. Code will be released.
E for details and hyperparameters for the
5.1 EXACT EQUIVALENCE WHENN USING CLASSIFIER ACTOR CRITIC
Our first experiment aims to validate our theoretical result under the required assumptions: when using classifier actor-critic as the RL algorithm, and when using tabular environment. We use 5 x 5 deter- ministic gridworld with 5 actions upp/oowa/lef//righh/nnotnn
We describe the reward function and other details in Appendix[E [E] To ensure that critic regularization converges to fixed point and avoid oscillatory learning dynamics, we update the policy using an exponential moving average. We also in- clude (unregularized) classifier actor-critic to confirm that regularization is important in some settings.
We compare these three methods in three environments. The first setting (Fig. 2 (left)) checks our theory that one-step RL and critic regularization should obtain the same policy. The second setting (Fig.
2 (center) shows that one-step RL and critic regularization learn the same (suboptimal) policy settings where using the Q-values for the behavioral policy lead suboptimal policy. The final setting is designed so that regularization increases the expected returns. The dataset is a single trajectory from the mitiil state to the goal. With such limited data, unregularized classifier actor critic overestimates the Q-values unseen actions, learning policy that mistakenly takes these actions. In contrast, the regularized approaches learn to imitate expert trajectory. Fig. (right) shows that both forms of regularization produce the optimal policy. In summary, these tabular experiments validate our thcoretical results, including in settings where regularization is useful and harmful.
5.2 PRACTICAL IMPLEMENTATIONS EXIIBIT SIMILAR BEIIAVIOR
Based on our theoretical analysis, predict that practical implementations of one-step RL and critic regularization will exhibit similar behavior, for a certain critic regularization coefficient. This section studies the tabular setting, and the following section will continuous control benchmark. For critic regularization, we used CQL (Kumar et al.] 2020) together with soft value iteration; following (Brandfonbrener et al. 2021), we implement one-step RL (reverse KL) using Q-learning
We designed a deterministic gridworld so one-step RL would fail to learn the optimal policy (see Fig. 3 (left)). If CQL. interpolates between the hehavioral policy (random) and the optimal policy,
7

---

number of rannoom MDPS
better than chance
(% same action]
1.0
20-
similarity
0.6
0.4
0.2
chance
0.0
0.0
0.5
1.0
10-2
10-1
10
10
10
fraction of suboptimal actionns where CQL and one agree
-llcmmnng
Behavioral
(CQL coefficient)
Figure 4: CQL. and one-step RI. take similar actions on most MDPs that resemble Fig-
Figure 5: One-step RL is most similar to CQL with a
moderate regularization coefficient.
then the argmax action would always be the same as the action for π² Based on our analysis, we make different prediction: that CQL will Iearn policy similar to the one-step RL policy. We show results in Fig. 3 (right), just showing the argmax action for visual clarity. The CQL policy takes actions away from both the high-reward state and the low reward state, similar the behavioral policy but different from both the behavioral policy and the optimal policy. This experiment suggests that CQL can exhibit behavior similar to one-step RL. Of course, this effect is mediated by the regularization strength: a larger regularization coefficient would cause CQL to learn random policy, and a coefficient of O would make CQL identical Q-learning.
How often does one-step RL approximate CQL? To show that the results in Fig. 3 are not cherry-picked, we repeated this experiment using 100 MDPs that are structurally similar to that in Fig. B] but where the locations of the high-reward and low reward state are randomized. In each randomly generated MDP, we determine whether CQL exhibits behavior similar to one-step RL by looking at the states where CQL. takes actions that differ from the reward-maximinn reward-maximizing maximiziniiii actions (as determined by rumning Q-learning with unlimited data). Since there are five total actions, a random policy would have a similarity score of 20%. As shown in Fig. 4 the similarity score is significantly higher than chance for the vast majority of MDPs, showing that one-step RL and CQL() 10) produce similar policies on most such gridworlds.
When does one-step RL approximate CQL? Because one-step RL. is highly regularized (policy iteration is truncated after just one step), one might imagine that it would be most similar to CQL with very large regularization coefficient. To study this, we use the same environment (Fig. 33 and measure the fraction of states where one-step RL and CQL choose the same argmax action. As shown in Fig. 5 one-step RL is most similar to CQL with moderate regularization (A = 10), and is less similar CQL with a very strong regularization.
5.3 TESTING PREDICTIONS ABOUT EXISTING OFFLINE RL METHODS
Our final set of experiments studies whether our theoretical results can make accurate testable predictions about practically-used regularization methods in a setting where they commonly used: offline RI. benchmarks with continuous states and actions. For these experiments, we will use well-tuned implementations of CQL and one-step RL from Hoffman et al. al.] (2020), using the default hyperparameters without modification. We made one change to the one-step RL implementation makethe comparison more fair: because CQL learns two Q functions and takes the minimum (a trick introduced in Fujimoto et al. (2018)), we applied this same parametrization to the one-step RL implementation. Since offline RL methods can perform different on datasets of varying quality (Wang et al. 2020; Fujimoto & Gul [021] Paine et al., 2020 Wang et al. [2021] Fujimoto et al. 2019), we will repeat our experiments on four datasets from the D4RI. benchmark (Fu et al. 2020).
Lower bounds on Q-values. One oft-cited benefit of critic regularization that it has guarantees about palue-estimatioo (Kumar et al. 2020): under appropriate assumptions, the learned value function will underestimate the discounted expected returns of the policy. Because our analysis shows conenction between one-step RL and critic regularization, it raises the question of whether one-step RL methods have similar value-cstimation properties. Taken at face value, this hypothesis seems obvious: the behavioral critic estimates the value of the behavioral policy, it should underestimate the value of any policy that is better than the behavioral policy. Despite this, the lower bound property of methods like one-step RL rarely discussed, suggesting that it has yet to be widely appreciated.
Fig. 6 shows both these predicted and actual (discounted) returns throughout the course of training. The results for one-step RI. confirm our theoretical prediction on 4/4 datasets: the Q-values from
8

---

medium-expert
medium+eplay
medium
random
1 1-step RL
overestimatiin
predicted returns actual returns
returs
Soa
underestimation
CQL
overastimation
predicted returns actual returns
returns
500
underestimation
0.5
10
0.5
0.5
gradient steps
146
gradient steps
1e6
gradient steps
gradient steps
Figure 6: Q-value under/over-sstimatinn. (Top) Experiments on benchmark datasets varying quality show
that one-step RI underestimates the Q-values. (Bottom) Despite the theoretical guarantees about critic regular- ization (CQL) yielding underestimates, in practice sometimes overestimate the actual returns. We plot the and standard deviation across five random seeds. Note that Q-values are equivalent to the value function
ohserve that the values learned via critic regularization
medium-expert
medium-reppay
medium
random
0.15
MSE
0.2
0.35
0.10
1.0
0.1
0.05
gradient steps 1c6
gradient steps 166
gradient steps
1e6
gradient steps
1e6
Critic regularization causes actor regularization. Performing critic regularization via CQL implicitly results actor regularization, similar to one-step RL; the MSE between the predicted actions and the dataset actions decreases. We plot mean and standard deviation across five random seeds.
Figure 7:
one-step RL underestimate the actual returns. In contrast, we observe that critic regularization overestimates the true returns on 2/4 environments, perhaps because the regularization coefficients used to achieve good returns in practice are too weak to guarantee the lower bound property, and perhaps because the theoretical guarantees are only guaranteed to hold at convergence. these experiments confirm our theoretical predictions that one-step RL will result in Q-values that are underestimations, while also questioning the claim that critic regularization methods are always preferable for ensuring underestimation.
total,
Critic regularization causes actor regularization. Our analysis in Sec. Sec. 4 not only suggests that one-step RL methods might inherit properties of critic regularization (as studied in the previous section), but also suggests that critic regularization methods may behave like one-step methods. In particular, while critic regularization methods such as CQL do not explicitly regularize their actor, hypothesize that they implicitly regularize the actor (Lemma 4.2) similar to how one-step RL methods explicitly regularize the actor.
We measure the MSE between the action in the dataset and the action predicted by the learned policy, Fig. 7 shows the results. Some CQL implementations, including ours, "warm-start" the actor by applying behavioral cloning loss for 50,000 iterations; our plots so that any effect is caused solely by the critic regularization. On 44 datasets, we observe that the MSE between the CQL policy's actions and the actions in the datasets decreases throughout training. Perhaps the one exception is on the medi um um-uuplaa replay dataset, where the MSE eventually starts increase after 5e5 gradient steps. While directly regularizing the actor leads MSE errors that are ~ 3x smaller, this plot nevertheless provides evidence that critic regularization indirectly regularizes the actor.
omit these initial gradient steps from
CONCLUSION
In this paper, we drew a connection between two seemingly-distonn -iitint RL regularization methods: one-step RL and critic regularization. While our analysis made assumptions that typically violated in practice, it nonetheless made accurate, testable predictions about practical methods with commonly- used hyperparameters: critic regularization methods can behave like one-step methods, and vice versa.

---

REPRODUCIBILITY STATEMENT
We have included full experimental details Appendix and proof' in Appendix
REFERENCES
Shoshana Ahramovich and Lars-Erik Persson. Some new estimates of the 'Jensen gap" Journal of Inequalities and Applications, 2016(1):1-9, 2016.
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, 2019.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified Q-ensemble. Advances in neural information processing systems. 34:7436-7447, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pes- simistic bootstrapping for uncertainty-drive offline reinforcement learning, arXiv preprint 22Xiv:200..11555 2022.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline RI without off- off-policy cvaluation. Advances Neural Information Processing Systems, 34:4933-4946, 2021.
Jacob Buckman, Carles Gelada, and Marc Bellemare. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint arXiv:2009.06799 2020.
Roberto Calandra, Andrew Owens, Manu Upadhyaya, Wenzhen Yuan, Justin Lin, Edward Adelson, and Sergey Levine. The feeling of success: Does touch sensing help predict grasp outcomes? arXiv preprint 1rXiv:1710.05512. 2017.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, al. Actionable models: Unsupervised offline reinforcement learning of robotic skills, arXiv preprint 104.v:20040077499 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Picter Abbcel, Aravind Srinivas, and Igor Mordatch. Decision transformer. Reinforcement learning via sequence modeling. arXiv preprint ar0iv:2106.00000000 01345, 2021.
Yiming Ding, Carlos Florensa, Pieter Abheel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rys: What is essential for offline rl via supervised learning? In International Conference Learning Representations, 2021.
Benjamin Eysenbach, Xinyang Geng, Sergey Levine, and Ruslan Salakhutdinov. Rewriting history with inverse rl: Hindsight inference policy improvement. Advances in Neural Information Processing Systems, 2020a.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-lcaming: Learning to achieve goals via recursive classification. In International Conference on Learning Representations, 2020b.
Benjamin Eysenbach, Sergey Levine, and Ruslan Salakhutdinov. Replacing rewards with examples: Example- based policy search via recursive classification. Advances in Neural Information Processing Systems, 34: 11541-11552, 2021.
Benjamin Eysenbach, Soumith Udatha, Sergey Levine, and Ruslan Salakhutdinov, Imitating past successes be very suboptimal. arXiv preprint arXiv:2206.0344 2022.
Chelsea Finn. Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pp. 49-58. PMLR, 2016.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine, Variational inverse control with events: general framework for data driven reward definition. Advances in neural information processing systems, 31, 2018.
Justin Fu, Aviral Kumar, Ofr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.077 072/9 2020.
10

---

Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 14-20132-20144 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation emoor in actor-critic methods. In International conference on machine learning. pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference machine learning, pp. 2052 2062. PMLR, 2019.
Xiang Gao, Meera Sitharam, and Adrian E Roitberg. Bounds the Jensen gap, and implications for mean- concentrated distributions. arXiv preprint arXiv://12.05267. 2017.
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. theory of regularized Markov decision processes. In International Conference on Machine Learning, pp. 2160021661 PMLR, 2019.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning reach goals via iterated supervised learning. International Conference on Learning Reprexentations, 2020.
Çaglar Gülçchre, Ziyu Wang, Alexander Novikov, Tom Paine, Sergio Gómez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RL unplugged: Benchmarks for offline reinforcement learning. 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abheel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp.
PMLR, 2018.
Kyle Hatch, Tianhe Yu, Rafael Rafailov, and Chelsea Finn. Example-based offline reinforcement learning without rewards. Proceedings of Machine Learning Research vol, 144:1-17, 2022.
Flad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference Machine Learning, pp. 2681-2691. PMLR, 2019.
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang. Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie, Ziyu Wang. Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint URL[attps: / aaxivv.ivv..../.v../...a./a...v. org/ abs 2000 00979
arXiv:20066. 00979, 2020.
Herbert Edwin Huntley. Dimensional analyxis. Dover publications, 1967.
Dmitry Kalashmikov, Alex Irpan. Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishman, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pp. 651-673. PMLR, 2018.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt- Continuous multi-task robotic reinforcement learning scale. arXiv preprint arXiv:2104.00 08212, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. In International Conference on Learning Representations, 2021.
Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint 132. 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforce- learning. Advances in Neural Information Processing Systems, 33:1 179-199919, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning. pp. 45-73. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Alexander C Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. arXiv preprint arXiv 11700 2020.
Cong Lu, Philip Ball, Jack Parker-Holde Michael Oshorne, and Stephen J Roberts. Revisiting design choices in offlin model based reinforcement learning. In International Conference Learning Representations, 2021.
11

---

Corey Lynch and Pieme Sermanet. Language conditioned imitation leamingg over unstructured data. Proceedings of Robotics: Science and Systems, 2021.
Jiafei Lyu, Xiaoteng Ma. Jiangpeng Yan, and Xiu Li. Efficient continuous control with double actors and regularized critics. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7655- 7663, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning, ArXiv, Abss/312.56000 2013.
Ofir Nachum, Bo Dai, llya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint 1rXiv:1912.02074, 2019.
John Nash. Non-cooperative games. Annals of mathematics, pp. 286-295, 1951.
Gergely Neu, Anders Jonsson. and Vicenç Gómez. unified view sutropy-regularize Markov decision processes, arXiv preprint XXiv:1705.077999 2017.
Tom Le Paine.
Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055 2020.
Keiran Paster, Sheila A Mcllraith, and Jimmy Ba. Planning from pixels using inverse dynamics models. arXiv preprint arXiv:2012.02411 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-wcighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 4rXiv:1910.00177, 2019.
Jan Peters and Stefan Schaal. Reinforcement learning by reward weight-eeghhted regression for operational space control. In Proceedings of the 24th international conference Machine learning, pp. 745-750, 2007.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010.
Lerrel Pinto and Abhinav Gupta. Supersizing sulf-superii Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference robotics and automation (ICRA), pp. 3406 3413, IEEE, 2016.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi- Semi-parametric topological memory for navigation. In International Conference on Learning Representations, 2018.
Bruno Scherrer. Improved and generalized upper bounds the complexity of policy iteration. Advances in Neural Information Processing Systems, 26, 2013.
Noah Siegel, Jost Tobias Springenherg, Felix Berkenkamp, Abbas Ahdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396, 2020.
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering, arXiv preprint
2019.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaškowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv: 1912.02877 2019.
Hao Sun, Zhizhong Li, Xiaotong Liu, Bolei Zhou, and Dahua Lin. Policy continuation with hindsight inverse dynamics, Advances in Neural Information Processing Systems, 32, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Hsiao-Yu Yu Tung, Adam W Harley, Liang-Kang, Huang, and Katerina Fragkiadaki. Reward learning from narrated demonstrations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 7004-7013, 2018
Adam Villaflor, John Dolan, and leff Schneider, Fine-tuning offline reinforcement learning with model- based policy optimization. Offline Reinforcement Learning Workshop at Neurul Information Processing Systems, 2020.
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, al. Exponentially weighted imitation learning for batched historical data. Advances Neural Information Processing Systems, 31, 2018.
12

---

Ruosong Wang. Yifan Wu, Ruslan Salakhutdinov. and Sham Kakade. Instabilities of offline rl with pre-trained neural representation. In International Conference on Machine Learning, pp.
PMLR, 2021.
Ziyu Wang. Alexander Novikov, Konrad Zolna, Josh Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel. Caglar Gulcehre. Nicolas Heess, al. Critic regularized regression. Advances Neural Information Processing Systems, 33:7768-7778, 2020.
Junfeng Wen, Saurabh Kumar, Ramki Gummadi, and Dale Schuurmans. Characterizing the gap between actor-critic and policy gradient. In International Conference on Machine Learning, pp. 11101-11111 PMLR, 2021.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:191..116/1 2019.
Rui Yang. Meng Fang. Lei Han, Yali Du, Feng Luo, and Xiu Li. MHER: Model-based hindsight experience replay. ArXiv, abs/2107.an./ 107.00306, 2021.
Tianhe Yu, Aviral Kumar, Rafacl Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative oftline model-bases policy optimization. Advances in neural information processing systems, -4926754-28967, 2021.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforvement learning. arXiv preprint 20Xtv:2011.0721.1 2020.
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience. arxiv preprint 1. 2020.
13

---

APPENDICES
In the Appendices, we provide proofs of the theoretical results (Appendix [ extend the analysis to other RL settings (Appendices C.D). and then provide details of the experiments (Appendix [E)
A PROOFS
A.1 PROOF OF LEMMA 4.1
Proof sketch. Lemma shows that classifier actor critic converges. The key idea of the proof will be show that the incremental updates for classifier actor critic are exactly the same the incremental updates for Q-learning. Q-learning converges, so algorithm that performs the same incremental updates Q-learning must also converge.
Proof. As the cross entropy loss is minimized when the predictions equal the labels, updates for (s,a) Q[s,a)+I & y=.2t(2.a)=T If the updates are performed by averaging over all possible next states (e.g., in the tabular setting), these updates are equivalent to directly updating Q(x, a) < (). = r(s, a) "BBp(eees..)rrarie [Qr(s', a')], which is the standard policy evaluation update for policy (a 8). Thus, we can invoke the standard result that policy evaluation converges to Q" (Agarwal et al. 2019 Theorem 1.14.) to argue that updates for Leniii likewise converge to Q".
Cemne (Q, T) can be written as Q(s.a)
In this proof, the TD targets were the expectation over the state and next action. If Eq. were optimized using a single-sample estimate of this expectation, y r(s, 7 d'), then the updates would be biased:
(s.
y y y
E
y (s, y (s) +1
@ a)
In settings with stochastic transitions or policies, these updates would result in estimating lower bound on Q" (s,a) a).
A.2 PROOF OF LEMMA 4.2 AND THEBOREM
Proof. Our proof proceeds in three steps. First, we derive the update equations for the regularized critic update. That is, we maintained a table Q-values, what would the new value for Q(s, a) be? Second, show that these updates are equivalent performing policy evaluation on re- parametrized critic (ss, a) Qss, a) r(als)
We invoke the standard results for policy evaluation
to prove convergence that Q(s,, a) convergences. Finally, we undo the reparametrization to obtain convergence results for ((s, a).
Step O. We start by rearranging the regularized critic objective:
Co (On yf 2. & Loino (Q, y T.R1) + Eppco)rals) (log(Q(s, + 1)] - Ep(oss(ais) [og(((ss,) + +1)]
= Epta,o y (x log a)s
Q(s,a) + log Q(s,a)+1
(Bplosraals (1o((Q(s,)) + 1)] - Ep( [lon(Q(s 1)]
-pEcoo)) (0 (s, log a(s,a) +I [(x,))
((s,a) a)
log
furg June Q(+,a) +1
Bp(s)a(aaa) log
Q(5,a)
= -Epcooo) (s log a(+,))+1
Q(s,a)
+ Ep(c))(ala)
log Q(s,a)+1
For the cancelation the third line, used the fact that p(s. p(s)B(a p)ps))]()))(( 8).
14

---

Step 1. To start, note that the regularized critic update is equivalent weighted classification loss: positive examples are sampled (s, a) ~ p(s)3(a s) and receive weight 9 (s.a)
(x.0- and negative
examples are sampled (s, a) ~ p(s) s) and receive weight y (x, classifier is given by
The Bayes' optimal
R. (T. (c,a)) P(3)(() ( P(
Q(s. a) ((s,a)
y Q (s. (s (s. 8) x(a
g( p(s) s)
Solving for (s,, a) on the left hand side, the optimal value for Q(s, a) is given by
3(a s) = (r(s,a) a) + (aa a')l) 3(a T(a s)
Q(s, a) = yr 0m (s.
(8)
x(a 8)
This equation tells what each update for the regularized critic loss does.
Step 2. To analyze these updates, we define ((s, a) A Qe(s, a)
s(als) M(aas)"
Then these updates can be
written using (ss, a) as
3(a s) T(a 8)
(((s,)= (p(rlka))) - @ d 3(a' [') s's B(a ss)
(1,, a)
(9)
Taa (s)
x(a s)
which can be simplified to
Q r(s, a) + +
[ă(s' a')
(10)
Note that the ratio S(a'(s) (aals") inside the expectation acts like an importance weight, so that expectation over (a' s') becomes an expectation over B(a' s'). Thus, the regularized critic updates are equivalent to perform policy evaluation on ((s, An immediately consequence is that the regularized critic updates converge, and they converge to (" a) Q (s,
Step 3. Finally, translate these convergence results for (s, into convergence results for Q(s, a). Written in terms the original Q-values, we see that the optimal critic for the regularized critic update is
Q' a) = (s, a) B(a s)
3(a a s (
Taa Q=
(11)
This completes the proof of Lemma 4.2
We now prove Theorem 4.3 by applying logarithm:
Proof.
ls) log (Q)(s.))
8(a 8) T(a 8)
log (og + log 3(a s) log ( s).
We note that our proof does not account for stochastic and function approximation errors. However, we assume that the TD updates are deterministic (e.g., as they are deterministic MDPs), then the updates for classifier actor-critic are identical to those of Q-learning (Lemma 4.1.. Thus, it immediately inherits any theoretical results regarding the propagation of errors Q-learning.
While this Theorem 4.3 shows that one-step RL and critic regularization have the same fixed point, it does not say how many transitions or gradient updates are required reach those fixed points.
15

---

A.3 WHY USE THE CROSS-ENTROPY LOSS?
Our proof Theorem 4.3 helps explain why classifier actor-critic use the cross entropy loss for the critic loss, rather than the MSE loss. Precisely, our analysis requires that the optimal Q function be ratio, Q(s a) (( The cross entropy loss can readily estimate ratios. For example,
B[a[s]
the optimal classifier for data drawn from p(z) and ((x) ) C(x) p(=)+q(z)) so the ratio can be C(x) z(æ) However, fitting a function ((x) to data drawn from (say) 1:1 mixture
p(x)
expressed as C-C(#) q(a)"
of (((x) and ((x)) would result in C(x) }p(x)) { which cannot transform express the
ratio p(r) as function of C(z).
((2)
A.4 VALIDATING THE THEORY
Our theoretical results suggest that one-step RL and critic regularization should be most simi- lar with critic regularization is applied with regularization coefficient of À 1. To test this hypothesis, we took the task from Fig. (Left)
fraction of same actions
0.8
and measured the similarity between one-step RL and critic-regularized classifier actor critic, for varying values of the critic regularization parameter. We measured the similarity of the policies obtained by the methods by count-
0.4
10
10
10a
regularization coefficient (A)
ing the fraction of states where the two methods choose the same (argmax) action. The results, shown in Fig. g validate our theoretical predic- tion that these methods should be most similar with À = 1.
Figure 8: Under the assumptions of Theorem 4.3
one-
step RL is most similar to critic regularization with a coefficient of À 1.
A.5 WHAT ABOUT USING THE POLICY GRADIENT?
Our analysis fundamentally requires using TD learning: the key step is that doing TD backups using one policy is equivalent to doing (modified) TD backups with different policy. However, the actor updates for both methods could be implemented using a policy gradient or natural gradient, rather than straight-through gradient estimator. Indeed, much of the work one-step RL methods (Peng et al.] [2199] Sicgel et al. 2020) uses an actor update that resembles policy gradient or natural policy gradient (e.g., 1-step RL with a reverse KL penalty (Brandfonbrener al. 2021
B VARYING THE REGULARIZATION COEFFICIENT
While our main analysis (Theorem 4.3 showed that regularization and critic regularization yield the same policy when these regularizers applied with certain strength, in practice the strength of regularization controlled by hyperparameter. This hyperparameter raises a question: does the connection between one-step RL and critic regularization hold for different values of this hyperpa- rameter?
In this section, we show that there remains precise connection between actor and critic regularization, even for different values of this hyperparameter. This result not only suggests that the connection is stronger than initially suggested by the main result. Proving this connection also helps highlight how many regularization methods can be cast from similar mold.
B. A REGULARIZATION COFFFICIENT.
We start by modifying the actor regularizer and critic regularizer introduced in Sec. 3.4 to include an additional hyperparameter.
16

---

Mixture policy. Both the actor and critic losses will make use of mixture policy, (1 A) s) + A3(a 8), where À E [0,1] will be a hyperparameter. Larger values of À yield a mixture policy that is closer to the behavioral policy; this will correspond to higher degrees of regularization. Mixtures of policies are commonly used in practice (Kumar et al. 2020) Appendix F), Villaflor et al. 2020 Eq. 11), (Finn et al. 2016, Sec. 4.3) (Lyu et al.] (022) (Hazan et al. 2019 Eq. 2.5), even though it rarely appears in theoretical offline RL literature. Indeed, because critic regularization resembles a two-player zero-sum game, mixture policies might even be required to find a (Nash) equilibrium of the critic regularizer (Nash 1951).
A-weighted critic loss. With this concept of a mixture policy, we define the X-weighted actor and critic regularizers. For the X- X-weighted critic loss, we will change how the TD targets are computed. Instead of sampling the next action from я or 3, we will sample the next action from a XTD-weighted combination of these two policies, reminiscent of how prior work has regularized the actions sampled for the TD backup (Fujimoto et al. 2019; Zhou et al. 2020
y A A (1-A))++B (s,a) r(s, + 7
[Q(s',
(s.)) (1 (-AAm) a) (mm
When introducing one-step RL in Sec. 3.4 we used ATD = 1.
Using this TD target, the X-weighted critic loss can now be written as combination of the un- regularized objective (Eq. 4 plus the regularized objective (
y y (s.a)
log Q(s.a)
ferai Acnos) _ (1 - Ausie))
(-Ep(a.))
ymm (s, +1 log a) +1
y y a1 Q a) +1
((s.a)+1
L (TTD(s, -1 log ((s.a) + () +1 log ((s,a)+1 +1
yAm (s.a)
1
+ X
-B p(ano)
(-(1)
y Am (s. log Q(s.a)
pla.a)
as. +1
((+ +1
ym +1 log
Q(
~11 Ammel (+ (-1x) Pmme
(12)
The second line rewrites this objective: the first term looks the same as the original "positive" term in the critic objective, while the "negative" term uses actions sampled from a mixture of the current policy and the behavioral policy. When Aentic = 1, we recover the regularized critic loss introduced in 3.c.
A- A-weighted actor loss. Finally, the strength of the actor regularizer can be controlled by changing the reverse KL. penalty. While it may seem like changing the reward scale would varying the strength of the actor loss, this not the case for classifier actor critic because of the log(-) in the actor loss. Instead, we will relax the reverse KL penalty between the learned policy át s) and the behavioral policy B(a s) so that only the mixture policy only needs to be close to behavioral policy:
Cisa AKa.) Epte)t(als) [log Q(s, a) + log B(a s) - log ((1 (T s) + Axa s))] (13)
As indicated on the second line, replacing ((a s) with the mixture policy has an effect similar to that of decreasing the weight applied to the KL penalty. The approximation on the second line is
determined by the Jensen Gap (Abramovich & Persson 2016 Gao et al. 2017). When introducing
one-step RL in Sec. 3.4 we used ÁKI. 1, together with ÀTD
In summary, the strength of the actor and critic regularizers can be controlled through additional hyperaarameters (Aaritic ÀTD, >Kt.) Indeed, it is typical for offline RL methods require many hyperparameters (Brandronbrener et al., [202T] Lu et al. 202T Paine et al., 20200) Wu et al. 2019). and performance sensitive to their settings. However, the close connection that we have shown between actor and critic regularizers allows us to decrease the number of hyperparameters.
B.2 ANALYSIS
In our main result (Thm. 4.3 we showed that one-stel RL and critic regularization are equivalent when Agmie AID ARL This is a large value for the regularization strength, and we now consider what happens for smaller degrees of regularization: is there still a connection between one-step RL and critic regularization?
17

---

The following theorem will prove that this is the case. In particular, applying critic regularization with coefficient Aemu yields the same policy as applying one-step RL with dyy = AKL = Aestic That is, there is very simple recipe for converting the hyperparameters for critic regularization into the hyperparameters for one-step RI..
Theorem B.1. Let policy 5(a s) be given, let Q a) be the Q-function of the behavioral policy, and let QAm a, Aerite) be the critic obtained by the Weinie-eighhted regularized critic update (Eq. 12) using TD targets (s, If Xeritic = XTD = AKL. then the AKL-wety weighted actor loss (Eq. 13) equivalent to the un-regularized policy objective using the regularized critic:
Ep(s)m(ala) [log Q(s.a) log B(a 8) log ((1 Ax)) s) ARn.(aa s)]]
= -E](a))) log O, (s, a, Aenme)
for all states s.
While we used the cross entropy loss for this result, it turns out that the result also holds for the more standard MSE loss (we omit the proof for brevity).
B we discuss few limitations of this result. Like the rest the analysis this paper, the form of critic regularizer is different from that often used in practice. Additionally, our analysis assumes ignores many sources of errors (e.g., sampling, function approximation), and assumes that each objective is optimized exactly.
Limitations. Before presenting the proof in Sec.
B.3 PROOF OF THEOREM B.1
Proof. We start by defining the fixed point of the X-weighted regularized critic loss. Like in the single-task setting, this loss resembles weighted classification problem, so we can write down the Bayes' optimal classifier as
ATD (s.a) (s.a)+) p(s)B(a
(ss,a) Q(s, a) +1
(mm()))
p(s)B(a 8) ) p(s)((( damie s) demeaaa s))
y (s.0)t
y (s 8)
y B(a 8) + (1 ) 8) + dermo s)
Solving for Q(s, a) on the left hand side, the optimal value for Q(s, a) is given by
B(a s)
Q(s, a) = Am (s, (1 Aoine (T 8)
3
3(a (8)
(r(s,a) +
a'))) (1 Aestie 8) Alane 3(e
(14)
Note that the next action a' is sampled from a mixture policy defined by ATD. This equation tells us what each update for the X-weighted regularized critic loss does.
To analyze these updates, we define
(=, a) ((s. (1 Aenis (a 8) + A
B(a |
Like before, the ratio [1-Am) Amm can act like an importance weight. When \TD = Aerities then this importance weight cancels with the sampling distribution, providing the following identity:
8(a's')
E belt
B(a | @(s..)) (1 Aetue \T s) Acaiic
=10 Am)
Ep(s" 12 a)).
Substituting this identity in Eq. 14 we can write the updates using Q(s, a): a) Q(s,a) (1 demie [aa 8) + Xemi 3(a s)
3(a s)
8(a = -(1(s,0) + +Bpee (1 (1 Aenie (a) 8) + Aeas: s)²
18

---

which can be simplified to
Q(s a) = r(s, a) + B 2f
We then translate these convergence results for Q(s, a) into convergence results for Q(s, a). Written in terms of the original Q-values, we see that the optimal critic for the regularized critic update is
3(a (s)
(s. Q2 (1 Aeritie (onniier((e (a s) + (
(15)
Note that this holds for any value of Acritie = ATD E [0, 1]. This result suggests that two common forms of regularization, decreasing the values predicted unseen actions and regularizing the actions used in the TD backup, can produce the same effect: critic that estimates the Q-values of the behavioral policy (multiplied by some importance weight).
Finally, substitute this Q-function into the un-regularized actor loss, we see that the result is equivalent to the A-weighted actor loss:
(p(o) [log Q', a)] -Ep(s)((aa)) log (s, + log B(a s) log ((1 )KL) s) AKL s))
X weeghhed actor regultzizer
C REGULARIZATION FOR GOAL-CONDITIONED PROBLEMS
Like single-task RL problems, coal-conditioned RL. problems have also been approached with both one-step methods (Ghosh et al.) 2020; Ding et al. 2019 Sun et al. 2019) and critic regular- ization (Chebotar et al. 2021). these problems, the aim is to learn a goal-conditioned policy [a s, 8g) that maximizes the expected discounted sum of goal-conditioned rewards r' a), where goals are sampled 89 ~ (sg))
Σ Errra(on
max E (m)((e))(((ll))))
We will use the goal-conditioned reward function (y,ss, a) p(s' = Sg s, a). which is defined in terms of the environment dynamics. settings with discrete states, maximizing this reward function is equivalent to maximizing the sparse indicator reward function (ry(s, a) = 1(s = s)).
In this section, we show that one-step RL and critic regularization are equivalent for a certain goal- conditioned actor-critic method. Unlike our analysis in the single-task setting, this analysis here uses an existing method, C-learning (Eysenbach et al. 2020b). C-learning is TD method that already makes use of the cross entropy loss for training the critic:
log Q(s, 8g 8') Q(s, a, Sy = 8²) +
+ Bppeiiial))) yd (s, log Q a, 8g))) Q(s a, sg) +
max (1 - - Q
1
+ -Eplaa))
log Q(8, a, 8g = 8) + +1
where Q. (s, a. sy) = Epts [Q(s', , sy)] serves the role of the TD target.
The first two terms increase the Q-values while the last term decreases the Q-values. The actor is updated maximize the Q-values. While this objective for the actor can be written in many ways, we will write it as maximizing a log ratio because it will allow us to draw precise equivalence between actor and critic regularization:
max E (xpc)))) [log Q(s, 8,)]
We will now consider variants of C-learning that incorporate actor and critic regularization.
19

---

One-step RL. We will consider variant of C-learning that resembles one-step RL (Brandfonbrener et al. 202T). The critic update will be similar to before, but the next-actions sampled for the TD updates will be sampled from the marginal behavioral policy:
log Q(s,a, 89 = 8¹) Q(s,0, s') +
Q(s, a, 88) + y B y y a, log ays +1
max (1 - Eptraa.)))
1 log Q(s, 8g
+ B (p(s
where g (8, a, 8n) reverse KL divergence:
[Qr(6', a', "',,,n)))) sz)]. The actor update will be modified to include
max L (s) [log ((s, a, $z)) + log 3(a | s) t (a s. s,
(16)
Note that we are regularizing the policy to be similar to the average behavioral policy, ,(a s). Compared to regularization towards goal-conditioned behavioral policy S( 8, 8g), this choice gives the policy additional flexibility: when trying to reach goal sg. it allowed take actions that were not taken by B(a) 8g), long as they were taken by the behavioral policy when trying to reach some other goal s/g-
Critic regularization. To regularize the critic, we will modify the "negative" term in the C-learning objective to use actions sampled from the policy:
log Qss, s')
max (1 -7)
(17)
Q(s,a, Sg = 5') +1
Q(s,a,8g)
+1 (ppc,)))aaaa
TT a, sy) log Q(s, 8g)
(18)
(s [log Q(s, a, sy) +1
(19)
C.I ANALYSIS FOR GOAL-CONDITIONED PROBLEMS
Like in the single-task setting, these two forms of regularization yield the same fixed points:
Theorem C.1. Let policy (a s, 8y) be given, let Q" 8g) be the Q-values for the marginal behavioral policy 3(a | 8) and let (7 8g) be the critic obtained by the regularized critic update (Eq, 19.. Then performing regularized policy updates (Eq. T6) using the behavioral critic is equivalent to the un- regularizee policy objective using the regularized critic:
E=(als,sa) [log 2 (s,a, + log 3(a s) log (aa s, sg)) (m(a)ss [log as) for all states s and goals Sg.
Proof. We start by determining the fixed point of critic-regularized C-learning. Like in the single-task setting, the C-learning objective resembles weighted-classfficttion problem, so we can write down the Bayes' optimal classifier as
Q(s.a.88) Qss, a, (g) +1 ((1 7)p(s' 8y 8, a) + 7p(s sy)y(
((1 7)p(s' s9 s.a) + >p(s==)))('s_..........
p(s,) | 8,)
Solving for Q(s, a, 8g) on the left hand side, the optimal value for Q(s, a, 8g) is given by 3(a s)
Q(s a. = ((1 - ()s''' Sg s. yp(s sy sy)u(s' ( $g))
π 8, 8g)
This tells us what each crittc-regularized C-learning update does.
To analyze these updates, we define Q(s,, a, sy) Q(s, a, s,) written using ((s, a, sy) as
s(alasee) Then these updates can be Male)
8(a [s) 8) = (11 (1-7))(ss =s, s, + Q06'' ds,s)) B(a' l(a|l'))
( 1)
3(a a)
(s, 8,) <(a s, s,)
aTaa s0)) a(a s,)
20

---

These updates can be simplified
(s, a, 8g) = - -1pppp((s== - - 8, + + (ā(s (((s",e d', (8)8
Like before, the ratio 3(a"(s")
>(a')=",ag) inside the expectation acts like an importance weight. Thus, the
regularized critic updates are equivalent to perform policy evaluation on Q(s,, a, 82). Note that this is estimating the probability that the average behavioral policy 3(a s) reaches goal syi this is not the probability that a goal-directed behavioral policy B(a 8, $g) reaches the goal.
Finally, we translate these convergence results for (s,, a, 8g) into convergence results for Q(s, a, sg). Written in terms of the original Q-values, see that the optimal critic for the regularized critic update is
B(a s) (2 8g) x(a 8, 8,88) a', kg) = (s, 8g) I(a 8, 8,8g)
B(a
Thus, critic regularization implicitly regularizes the actor objective so that it is the same objective as one-step RL:
Pp(s)) (rc)))asmm(()))aaaaaa [log Q" (x. a,
Ep.))ay [oo (0(-)-) (s, (s, log B(a log TT 8, (,,5)))
D REGULARIZATION FOR EXAMPLE-BASED CONTROL PROBLEMS
While specifying tasks in terms of reward functions standard for MDPs, it can be difficult for real-world applications of RL. So, prior work has looked at specifying tasks by goal states (as in the previous section) or sets of states representing good outcomes (Pinto & Guptal 2016; Tung al. 20188] Fu et al., (018). In addition to requiring more flexible and user-friend forms of task specification, these algorithms targeted at real-world applications often demand regularization. same way that prior coul-conditioned RL algorithms have employed critic regularization. too have prior example-base extend our analysis to regularization of an example- based control algorithm. Again, we will show that a certain form of critic regularization equivalent to regularizing the actor.
control algorithms (Singh et al. 2019; [Htch et al. 2022). In this section, we
We first define the problem of example-based control (Fu al. 2018). In these problems, the agent is given a small collection of states ~ (s). which are examples of successful outcomes. The aim is to learn policy क (a 8) that maximizes the probability of reaching success state:
max B. (r(()))))
Σ Y (s()))
Note that this objective function exactly equivalent reward-maximization problem, with reward function r(s, a) (st).
In this section, show that one-step RL and critic regularization are equivalent for certain example- based control algorithm. Unlike our analysis in the single-task setting, this analysis here uses an existing method, RCE (Eysenbach et al. 2021). RCE is a TD method that already makes use of the cross entropy loss for traming the enite::
max (1- log a) 1 Q(s,a)
((s,a) + log Q + Ep(a.a) (t (s, log a(s,a) +1
where y (s, a) = Ep(s [Q(s', a')] serves the role of the TD target. The first two terms increase the Q-values while the last term decreases the Q-values. The actor is updated maximize the Q-values. While this objective for the actor can be written in many ways, we will write it as maximizing a log ratio because will allow us draw precise equivalence between actor and critic regularization:
max (p(s) [log ((s, a)]
We will now consider variants of RCE that incorporate actor and critic regularization.
21

---

One-step RL. We will consider variant of RCE that resembles one-step RL (Brandfonbrener
al. 2.2T). The critic update will be similar before, but the next-actions sampled for the TD updates will be sampled from the behavioral policy:
max (1 - (A log Q(s,a) @(s.a) +
Q(s,a)
+ Pp(aa))
ny Rt (s. a) log ((s,a)
log (+.
Q
where g a) E (st'")s.)) QQ(s d'}]. The actor update will be modified to include reverse KL divergence:
max (()))())))) log Q(s, a) + log 3(a - xaa s)]
(20)
To regularize the critie, we will modify the "negative" term in the RCE objective to use actions sampled from the policy:
Critic regularization.
Q(3,0) (1 - ([=7))p(())dta)) log ((s.0)+1 +1 + (p(aa)) -((1b)))
Q(s,a)
Ty 0 log ((s,a)+1 +1 + log Q(s, ((s,a-)+11
+
(21)
D.1 ANALYSIS FOR EXAMPLE-BASE BASEDD CONTROL PROBLEMS
Like in the single-task setting, these two forms of regularization yield the same fixed points:
Theorem D.1. Let policy (a 8) be given, let Q (s, a) be the Q-values for the behavioral policy 3(a s) and let Q? (s, a) be the critic obtained by the regularized critic update (Eq. 21). Then performing regularized policy updates (Eq. 220 using the behavioral critic equivalent to the un-regularized policy objective using the regularized critic:
Ex((]s) [log º (8,a) + log B(a 8) log T(a) {)] = Erta 8) log Q7
for all states
Proof. We start by determining the fixed point of critic-regularized RCE. Like in the single-task setting, The RCE objective resembles a weighted-classsfication problem, so we can write down the Bayes' optimal classifier as
(ss.a) a(s.a)+ +1 ((1 - (s) (s (s.a))B(a s) x(a
Solving for Qss, a) the left hand side, the optimal value for Q(s, a) is given by <(a
3(a
Q(s, a) = ((1 (1)))).(s) + Ty (s a))
This tells us what each critic-regulrrized RCE update does. To analyze these updates, we define ()s,a) & Q(s. Blals) Then these updates can be written
s(als)
using Q(s, a) as
((a () a(s,a) la
(11 -1 (1--))p(ss) t [0(s), a') 8(a' { 3(a s)
a(aa ( a(a s)
These updates can be simplified to
((s = (s) + ( "Bp( (Q(s''aa)
Like before, the ratio 8(a' =²) inside the expectation acts like importance weight. Thus, the x(o² 5
regularized critic updates equivalent to perform policy evaluation @(s.
Finally, translate these convergence results for Q(s, a) into convergence results for Q(s, a). Written in terms of the original Q-values, we see that the optimal critic for the regularized critic update is
Q' = (s. 3(a 8) T[a s)
3(a xaa s)
Q
Thus, critic regularization implicitly regularizes the actor objective so that it is the same objective as one-step RL:
(p(s).r(als) [log Q"(s, (s, E(to). log (s,
(s. a) log B(a s) log <(a ss)
22

---

E EXPERIMENTAL DETAILS
E TABULAR EXPERIMENTS
Implementing critic regularization for classifier actor critic. The objective for critic regular- ization in contrastive actor critic (Eq. 7 is nontrivial to optimize because of the eyclic dependency between the policy and the critic: simply alternating between optimizing the actor and the critic does not converge. In our experiments, we update the critic using an exponential moving average of the policy, as proposed in Wen al. (202I). We found that this decision was sufficient for ensuring convergence. When applying CQL the tabular setting (Figures B and 4p, we did not do this because soft value iteration represents the policy implicitly in terms of the value function.
Fig. (left) The initial state and goal state are located in opposite corners. The reward function is +1 for reaching the goal and 0 otherwise. We use a dataset of 20 trajectories, 50 steps each, collected by a random policy. We use y 0.95 and train for 20k full-batch updates, using a learning rate of 1e-2. The Q table is randomly initialized using a standard normal distribution.
Fig. 2 (center) The initial state and goal state are located in adjacent corners. The goal state has a reward of +3.5, the states between the initial state and goal state have a reward +1, and all other states (including the initial state) have a reward of +2. We use a dataset of 20 trajectories, 50 steps each, collected by random policy. We use y 0.95 and train for 20k full-batch updates, using learning rate le-2. The Q table is randomly initialized using a standard normal distribution.
Fig. (right) The initial state and goal state are located in adjacent corners. The reward is +0.01 the goal state and O otherwise. We use dataset of trajectories with 10 steps, which traces the following path:
[[0, 0), (1, 0), (1, 1), (1, 2), (1,3), (1, (0. 1), (0, (0,4), (0,4))) (0,
We use 7 = 0.95 and train for 10k full-batch updates, using a learning rate of 1e-2. The Q table is randomly initialized using standard normal distribution.
F There is bad state (reward of 100) next to the optimal state (reward of +1), so the behavioral
policy navigates away from the optimal state. We generate 10 trajectories of length 100 from uniform random policy. We use 7 0.95 and train each method for 10k full-batch updates. The Q table is randomly initialized using a standard normal distribution. One-step RI. performs SARSA updates while CQL performs soft value iteration (as suggested in the CQL paper).
Fig. 4 low-reward state (without replacement). The datasets are generated in the same way.
We generate 100 random variants of Fig. by randomly sampling the high-reward state and
We use the same enviromment and dataset as in Fig. but train the CQL agent with varying values of X, each with 5 random seeds. We train the one-step RL agent for random seeds. For each point on the X axis of Fig. []] we compare compute 5 x 5 pairwise comparisons and report the mean and standard deviation.
Fig. 5
E.2 CONTINUOUS CONTROL EXPERIMENTS
used the implementation of one-step RL (reverse KL) and CQL provided by Hoffman et al. (2220). We choose this implementation because it is well tuned and uses similar hyperparameters for the two methods. As mentioned in the main text, the only change we made to the implementation was adding the twin-Q trick to one-step RL, such that it matched the critic architecture used by CQL. We did not change any of the other hyperparameters, including hyperparameters controlling the regularization strength.
For the experiments in Figures 6 and
23


Reviews:
[{"review_id": "vsAoxMhuWS_", "reviewer": "Reviewer_hMZj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper makes a connection between regularizing the critic and regularizing the actor in the actor-critic framework. Specifically, the authors show that these two are equivalent with a specific choice of the regularizing parameter.  Extensive numerical simulations are provided to justify the theoretical findings. ", "summary_of_the_review": "Overall I think this paper provides an interesting viewpoint on the connection between critic regularization and actor regularization, and the numerical simulations are convincing. ", "strength_and_weaknesses": "This paper is easy to follow and the result seems interesting. \n\n(1) For Theorem 4.3, since in practice there are stochastic errors and function approximation errors, showing that the ideal update equations are equivalent is not entirely satisfactory. It will be interesting to see how these errors propagate in these two types of algorithms.\n\n(2) Since multi-step critic regularization and one-step RL are equivalent, but multi-step critic regularization seems to be less sample efficient to implement, does this suggest that one should use one-step RL in practice?\n\n(3) In both cases, the authors focus on policy iteration type of policy update. What about using policy gradient or natural policy gradient type of policy update?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MVmV1hvJq2e", "reviewer": "Reviewer_3nPe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a theoretical and empirical analysis with the main goal showing the similarity between two main categories of algorithms, one-step methods and critic regularization-based methods. Specifically, the paper shows that, under some conditions, the two types of methods are theoretically equivalent; it also shows that, in relevant practical cases, they also behave similarly in practice.", "summary_of_the_review": "In summary, I found this to be a good paper that can be of general interest for people working with offline RL algorithms of various kinds, and I thus recommend acceptance.\n\n-----\n_After rebuttal phase_: after reading the discussion with the area chair and the other reviewers, I now share their concerns on the significance and technical soundness of the results and I have thus lowered my score.", "strength_and_weaknesses": "Strengths:\n- The writing is very clear and the presentation of the content is easy to follow;\n- The theoretical results are compact and clear, and also well explained in the context of the full paper;\n- The experimental analysis is remarkably clear as well (I really like visualization in Figure 4 and Figure 5);\n- The extended analysis for other settings (e.g., goal-conditioned RL) reported in Appendix adds interesting material to the paper.\n\nWeaknesses:\n- The theoretical analysis a bit limited by the \"classifier actor-critic\" setting; it would be nice to explain in more detail why an analysis in the regression case is especially difficult to help the reader understand this choice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Jg80aDfSmUe", "reviewer": "Reviewer_pLRb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to draw a connection between the 'one-step' offline RL algorithms and the critic regularization-based ones. With some modification in actor or critic objective, the authors prove the same convergence results for the both methods in tabular settings. And the numerical simulations and empirical results on offline RL benchmark show the close connection between the both algorithms.", "summary_of_the_review": "This paper shows that the objectives of one step RL and CQL are equivalent. However, since the main results ignore the difference in optimization methods,  these algorithms actually have big difference. I think that the paper should mention this.", "strength_and_weaknesses": "Strength:\n\n1. this paper tries to reveal the relationship between 'one-step'-based and critic regularization-based offline RL methods, and provide a new perspective on the understanding of various offline RL algorithms.\n\n\nWeaknesses:\n1. Though this paper proves that both the offline methods can have the same convergence results, some preconditions should be satisfied and some key changes seems strange.\n2. The main results, i.e., thm 4.3 says that \"the critic regularization results in the same policy as one-step RL\" , which is doubtful. Actually, this theorem only shows that their objective is equivalent, not necessarily the algorithm. We know that different algorithms will generally results in quite different policy, which is essentially the key motivation behind one-step RL: all the value-based methods are optimizing the Bellman equation in some sense, but multiple step or iterative type algorithms are not neccearily better than the single step ones.  In particular, for critic regularization algorithm such as CQL, it is their optimizing method that defines what the algorithm really is - in this case it is just an iterative type value-based algorithm, and hence may suffer from the over use of the estimated behavior Q function, no matter its objective is equivalent to that of one-step RL or not.  Due to the above reason, CQL may not results in the same policy as one-step RL.\n\n3. Some of the experimental results require more detailed explanation, (e.g. numerical simulation results in Figure.2 make readers confusing). And there is a minor disagreement on the coefficient choice for both theoretical and empiciral results.\n3. It's better to provide the performance of both methods for more sufficient comparison.\n\n4. for eq.(3) , although the range of the normalized Q function and y function lies in (0,1), their integration is not one and hence not probability function. So what's the justification to compare them with cross entropy?\n\n5. About the coefficient $\\lambda$, the theoretical analysis suggests that $\\lambda=1$ can lead to the same convergence for both methods, while the experiments show both methods have the most similar results (Fig.5) if $\\lambda=10$. Maybe more analysis on various coefficients should be carried out.\n\n6. About the lower bounds on Q-values(Figure.6), the original paper proves that the complete CQL objective can lead to the underestimation on V value while not Q value, so I'm not sure it's appropriate to compare Q values.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "45XVEfIs5ms", "reviewer": "Reviewer_TugW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a theoretical analysis, hoping to show that one-step RL is equivalent to a certain type of critic regularization. ", "summary_of_the_review": "- The introduction is hard to follow. Particularly, your definition of regularization is different from what a normal RL reader may expect. Most of the methods cited in the introduction to combat over-estimation of offline RL are not normally known as *regularization* techniques. I would strongly suggest starting with a clear definition of what “you” mean by regularization before even start citing other papers and compare various method.\n\n- Last paragraph of the introduction is very unclear, likewise the contributions part. Clearly convey the contributions in short and informative statements. \n\n- Second paragraph of related work: Not sure how imitation learning (IL) falls into these categories. IL normally tries to generalize the seen actions at seen states, with no regard to maximizing expected return. That is, IL tries to clone the behavioural policy. In contrast, one step methods still have a policy improvement step, meaning that their resultant policy is likely to be better than the behavioural policy and it does not necessarily clone that. \n\n- Your Figure 1 seems to have not been referred to at all in the text. \n\n- Section 3.1: “… we assume rewards are positive, adding a positive constant to all rewards …” this is only correct if there is no terminal state, otherwise the values of terminal states should also be shift from (by definition being) zero. If that’s the case (which seems to be from the following part), make it explicit that the MDP is not episodic and there is no termination. \n\n- Section 3.1: your definition of reward is indeed expected reward (over next state). Paying attention to small details help the reader to follow your main points better. \n\n- Section 3.1: your definition of $Q^\\pi$ is undiscounted, yet your reward is strictly positive. The summation simply diverges. \n\n- Page 4, first paragraph: “… TD targets $y(s, a)$ are not considered learnable (i.e., we would apply a stop-gradient operator)” -> what do you mean? So, you do not try to solve second line of equation (1)?\n\n- Page 4, last line -> this update is SARSA, not Q-learning. \n\n- Paragraph after equation 4: what do you mean by “in tabular settings”? Do you mean the standard SARSA? Same issue in Lemma 4.1. The proof of Lemma 4.1 does not look right since you have a nonlinear transformation which does not commute with the expectation and Jensen’s inequality applies. \n\n- Again, proof of Lemma 4.1: you said the update can be written as $\\frac{Q}{Q+1}\\leftarrow \\frac{y}{1+y}$. This is only the loss function (and not the update) which then *incrementally* changes the value of $Q$. You need to show that under incremental change, the whole algorithm converges. (See for example textbooks on stochastic approximation, e.g., chapter 4 of Neuro-Dynamic Programming, Bertsekas and Tsitsiklis, 1996)\n\n- Page 5, paragraph after equation 5 -> this also rings a bell in my mind. As an alternative to the cross-entropy, one may directly learn the transformed Q values. See this reference: https://arxiv.org/abs/2203.07171 \n\n- Lemma 4.1 and 4.2 -> You need to clarify the assumptions under which these results hold. \n", "strength_and_weaknesses": "Strength:\n\n- The general idea of discovering equivalence between methods, which look like arbitrary choices, is quite interesting (and can be useful as it clarifies false expectations for performance improvement if different variations are indeed equivalent).\n\nWeaknesses:\n\n- Poor presentation: it is not an easy paper to follow, especially definitions and assumptions are often unclear (aside from notational and conceptual mistakes here and there). \n\n- Some of the results are in question, particularly, switching regression to classification has been done through a nonlinear transformation. It is unclear if this transformation still guarantees convergence to the desired values.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject