The In-Sample Softmax for Offline Reinforcement Learning
Authors: Chenjun Xiao
Han Wang
Yangchen Pan
Adam White
Martha White
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Offline Reinforcement Learning

Abstract: Reinforcement learning (RL) agents can leverage batches of previously collected data to extract a reasonable control policy. An emerging issue in this offline RL setting, however, is that the bootstrapping update underlying many of our methods suffers from insufficient action-coverage: standard max operator may select a maximal action that has not been seen in the dataset. Bootstrapping from these inaccurate values can lead to overestimation and even divergence. There are a growing number of methods that attempt to approximate an in-sample max, that only uses actions well-covered by the dataset. We highlight a simple fact: it is more straightforward to approximate an in-sample softmax using only actions in the dataset. We show that policy iteration based on the in-sample softmax converges, and that for decreasing temperatures it approaches the in-sample max. We derive an In-Sample Actor-Critic (AC), using this in-sample softmax, and show that it is consistently better or comparable to existing offline RL methods, and is also well-suited to fine-tuning. We release the code at github.com/hwang-ua/inac_pytorch.

Published as a conference paper at ICLR 2023
THE IN-SAMPLE SOFTMAX FOR OFFLINE REINFORCE- MENT LEARNING
Chenjun Xia0*:1,2 Han Wang*.f.,, Yangchen Pant: Adam White", Martha Martha
White
Huawei Noah's 's Ark Lab
2 University of Alberta; Alherta Machine Intelligence Institute (Amii)
3 University of Oxford
ABSTRACT
Reinforcement learning (RL) agents can leverage batches of previously collected data to extract reasonable control policy. An emerging issue in this offline RL setting, however, that the bootstrapping update underlying many of our methods suffers from insufficient action-coverage. standard max operator may select a maximal action that has not been seen in the dataset. Bootstrapping from these inaccurate values can lead to overestimation and even divergence. There are a growing number of methods that attempt to approximate an in-sample max, that only uses actions well-covered by the dataset. We highlight a simple fact: it is more straightforward to approximate an in-sample softmax using only actions in the dataset. We show that policy iteration based the in-sample softmax converges, and that for decreasing temperatures it approaches the in-sample max. We derive In-Sample Actor-Critic (AC), using this in-sample softmax, and show that it is consistently better or comparable to existing offline RL methods, and is also well- suited to fine-tuning. We release the code at aithub.com/wwang. wwwpwwa/naaaanccccc.. -pytorcc.
INTRODUCTION
common goal in reinforcement learning (RL) is to learn a control policy from data. In the offline setting, the agent has access to batch of previously collected data. This data could have been gathered under a near-optimal behavior policy, from a mediocre policy, or a mixture of different policies (perhaps produced by several human operators). A key challenge is to be robust this data gathering distribution, since we often do not have control over data collection in some application settings. Most approaches in offline RL learn action-values, either through Q-learning updates- bootstrapping off of maximal action in the next state-or for actor-critic algorithms where the action-values are updated using temporal-differene (TD) learning updates evaluate the actor.
In either case, poor action coverage can interact poorly with bootstrapping, yielding bad performance. The action-value updates based on TD involves bootstrapping off an estimate values in the next state. This bootstrapping is problematic if the value is an overestimate, which is likely to occur when there are actions that are never sampled in a state (Fujimoto al., 2018; Kumar al., 2019; Fujimoto et al., 2019). When using a maximum over actions, this overestimate will be selected, pushing up the value of the current state and action. Such updates can lead to poor policies and instability (Fujimoto et al., 2018; Kumar et al., 2019; Fujimoto et al., 2019).
There are two main approaches in offline RL to handle this over-estimation issue. One direction constrains the learned policy to be similar to the dataset policy (Wu et al., 2019; Peng et al., 2020; Nair et al., 2021; Brandfonbrener et al., 2021: Fujimoto & Gu, 2021). A related idea is to constrain the stationary distribution of the learned policy to be similar to the data distribution (Yang et al., 2022). The challenge with both these approaches is that they rely on the dataset being generated by an expert near-optimal policy. When used on datasets from more suboptimal policies- like those commonly found in industry- they do not perform well (Kostrikov et al., 2022). The other approach bootstrap off pessimistic value estimates (Kidambi et al., 2020; Kumar et al., 2020; Kostrikov et al.,
'These authors contributed equally to this work Work was done while the author was at Huawei Noah's Ark Lab (chenjun, han8, amw8, wh ihhmemmee @ua berta ca; yangchen, Sannchen.paneeee www.png.nn...c.mmmmmom ac.

---

Published as a conference paper at ICLR 2023
2021; Yu et al., 2021; Jin et al., 2021; Xiao et al., 2021) and relatedly identify and reduce the influence of out-of- disiiiiiibuttion actions using ensembles (Kumar et al., 2019; Agarwal et al., 2020; Chasemipour et al., 2021; Wu et al., 2021; Yang al., 2021; Bai et al., 2022).
One simply strategy that has been more recently proposed is to constrain the set of actions considered for bootstrapping the support of the dataset D. In other words, if (D(aas) is the conditional action distribution underlying the dataset, then we use max."; ("')==") q(s', aa) instead of max,' q(s,, a'): constrained or in-sample max. This idea was first introduced for Batch-Constrained Q-learning (BCQ) (Fujimoto et al., 2019) in the tabular setting, with generative model used to approximate and sample TD (a|s) (Fujimoto et al., 2019; Zhou al., 2020; Wu al., 2022). Implicit Q-learning (IQL) (Kostrikov et al., 2022) was the first model-free approximation to use this in in-aamle max, with later modification to be less conservative (Ma et al., 2022). IQL. instead uses expectile regression, to push the action-values to predict upper expectiles that are (close) lower bound to the true maximum. The approach nicely avoids estimating TD. and empirically performs well. Using only actions in the dataset is beneficial, because it can approach is be difficult to properly constrain the support of the learned model for Tp and ensure it does not output sitor--distibbutions actions.
There are, however, a few limitations to IQL. The IQL solution depends the action distribution not just the support. In practice, we would expect IQL to perform poorly when the data distribution is skewed towards suboptimal actions in some states, pulling down the expectile regression targets. We find evidence for this in our experiments. Additionally, convergence is difficult to analyze because expectile regression does not have closed-form solution. One recent work showed that the Bellman operator underlying an expectile value learning algorithm is a contraction, but only for the setting with deterministic transitions (Ma et al., 2022).
In this work, we revisit how to directly use the in-sample max. Our key insight is simple: sampling under support constraints is more straightforward for the softmax, in the entropy-regularizee setting. We first define the in- in-sample softmax and show that it maintains the same contraction and convergence properties as the standard softmax. Further, we show that with a decreasing temperature (entropy) parameter, the in-sample softmax approaches the in- nampa max. This formulation, therefore, is both useful for those wishing to incorporate entropy- reguaaizaagllaiizataioo and to give a reasonable approximation to the in-sample max by selecting small temperature. We then show that we can obtain a policy update that relies primarily on sampling from the wataset-which is naturally in-sample-rather than requiring samples from estimate of TD. We conclude by showing that our resulting In In-sample Actor-critic algorithm consistently outperforms or matches existing methods, despite being a notably simpler method, in offline RI experiments with and without fine-tuning.
PROBLEM SETTING
In this section we outline the key issue of action-coverage in offline RL that we address in this work.
2.1 MARKOV DECISION PROCESS
We consider finite Markov Decision Process (MDP) determined by M = AS,,A, (Puterman, 2014), where& S is finite state space, A is finite action space, r: S x A + R and P : S x A + ASS) are the reward and transition functions.' 1 The value function specifies the future discounted total reward obtained by following a policy S A(A), v (s) E" r ss) = s] where we use E" to denote the expectation under the distribution induced by the interconnection of and the environment. The corresponding action-value function is v0t a) = r(s, a) + + lu (s')) There exists an optimal policy T² that maximizes the values for all states S. We use v" and q¹" to denote the optimal value functions. The optimal
1) is the discount factor,
value satisfies the Bellman optimality equation,
2 (s) = max r(s, a) + [e
= r(s, +
maxx" (s' (s'. (1)
In this work more specifically consider the entropy-regularized MDP setting- also called the maximum entropy where an entropy term is added to the reward to encourage the policy to
We use the standard notation A(X) to denote the set of probability distributions over finite set X.
2

---

Published as a conference paper at ICLR 2023
be stochastic. The maximum-entropy value function is defined as
00
(*()) = 00 (8) + (s,
(s, σ) =
=)
E
3 log (alg(()))) =
(2)
for temperature and H. the discounted entropy regularization. The corresponding maximum en action-value function is q" (s. r(s, a) + +P [ (s'),, with soft Bellman optimality equations similarly modified as described the next section. As r -> O, we recover the original value function definitions. The curropyrregularizee
setting has become widely used (Ziebart et al., 2008; Mnih et al., 2016; Nachum et al., 2017: Asadi & Littman, 2017: Haarnoja et al., 2018; Mei et al., 2019; Xiao et al., 2019), because it 1) encourages exploration (Ziebart et al., 2008), 2) often makes objectives more smooth (Mei et al., 2019), and 3) provides these improvements even with small temperatures that do not significantly bias the solution to the original MDP (Song et al., 2019).
2.2 OFFLINE REINFORCEMENT LEARNING
In this work, we consider the problem of learning an optimal decision making policy from a previously collected offline dataset D = {&i, Og, Fa. s - We assume that the data is generated by executing behavior policy Tp- Note that we do assume direct access to Tp. In offline RL, the learning algorithm can only learn from samples in this D without further interaction with the environment.
One primary issue in offline RL is that TD may not have full coverage over actions. Greedy decisions based on learned value q q² could be problematic, especially when the value is overestimate for out-of distiiibiitttoonnn actions (Fujimoto et al., 2019). To overcome this issue, one popular approach to constrain the learned policy to be similar to TD. such as by adding KL-divergence term: max, Eg E. π ((aas]((sss])))))_(]
a) T DxLL(TTT (T(ss) (-[))) for some 7 > 0. The optimal policy for this objective must be on the support of TT: the KL constraint makes sure x(als)
as long as (alss)) = 0. This optimal policy, with closed-form solution T ra(ass) (als)) exp(g(s, a)/T), /). is also guaranteed to be an improvement on TD. Many offline RL algorithms are based on this nice idea (Wu et al., 2019; Peng et al., 2020; Nair et al., 2021; Brandfonbrener et al., 2021; Fujimoto & Gu, 2021). This KL constraint, however, can result in poor x' when Tp is sub-optimal, confirmed both in previous studies (Kostrikov et al., 2022) and our experimental results.
The other strategy is to consider an in-sample policy optimization, maxa EEEA [(als)q(s, a). where I Tp indicates the support of is subset of πz. This approach more directly avoids selecting -fi-----iittiitt actions. Though a simple idea, approximating this with a simple algorithm has been elusive, as discussed above. The simplest idea estimate Tu TD and directly constrain the support by sampling candidate actions from Tw. as proposed for Batch-Constraine Q-learning (Fujimoto et al., 2019). This simple approach, however, may not avoid bootstrapping from out-of-sample actions due to the error in the estimate Tw-
Surprisingly, the small modification to the in-sample softmax (Section 3) has not yet been considered for offline RL. Yet, moving from the in-sample (hard) max to the in-sample softmax facilitates developing a simple algorithm, as we discuss the remainder of this work.
3 THE IN-SAMPLE SOFTMAX OPTIMALITY
This section introduces the in-sample softmax optimality that provides simple implementation of in- sample bootstrapping. We first describe the standard soft Bellman optimality equations, then the modification to consider in-sample bootstrapping. Our simple algorithm comes from stepping back and recognizing the utility of considering in-sample bootstrapping for the entropy-regularized setting rather than only for the hard-max.
The soft Bellman optimality equations for maximum-entropy RL use the softmax in place of the max,
q't (s. = r(s,a) + P(.ss))))) log Σ a (s'
Fujimoto& & Gu (2021) use a behavior cloning regularization (a(s) 2 where a is action in the dataset. We note it is exactly KL regularization under Gaussian parameterization with standard deviation. Brandfonbrener al. (2021) propose one-step policy improvement method: first learn the value of up, then directly train policy maximize the learned value. Thus this is indeed a behavior regularized approach.
3

---

Published as a conference paper at ICLR 2023
This comes from the fact that hard max with entropy regularization is (a EnEap(a)q(s,a))
rH( = r log ΣeEA el(s,)/7. As -> 0, softmax (log-sum) (lg-suum-xp)) approaches the max."
We can modify this update to restrict the softmax to the support of TD:
- - r(s, a)
log Σ
Top (d'aa))
(4)
d'v
We call Eq. (4) the in-sample softmax optimality equation. is interesting to note that we can use a simple reformulation that facilitates sampling the inner term. For any 9.
((s,a))T
Σ
molas))ppplas) (
amn (a =)
Σ
a:
(5) This reformulation does not perfectly remove the role of (als))) but is significantly reduced. The support is no longer constrained using and instead the values are simply shifted by this term involving TD. will use this strategy below to develop our algorithm.
E
There are few interesting facts to note about in-sample softmax. First, we can show that similarly to the standard maximum-entropy bootstrap (shown formally in Lemma 3), we have for any g
log
max STD
Σ (als)) +H))
CNTD (2
Though this outcome is intuitive, it is a nice property that restricting the support of the log pog-sum-expp maintains the same relationship to the maximum-entrop: update with the same support constraint. It extends this result for the soft Bellman optimality update to the setting with support constraint. From this perspective, in-sample softmax can also viewed as a tool for conservative exploration: exploring prevent getting stuck in local optima, while still being suspicious of what the data does not know. This especially important when g is learned value approximation.
Second, we can also obtain closed-form greedy policy using the above (shown formally in Lemma 3), which we call the in- sample softmax greedy policy: for any q.
q(s, a) - log T
TT T (als) x (alss) exp
This closed-form solution looks similar to the KL-regularized solution mentioned in Section 2.2, where x' is constrained to be similar to Tp. The only difference is the additional - -log RD term inside the exponential. This small difference, however, has a big impact. allows the resulting policy to deviate much more from TD. In fact, because exp(- lop(-logTD(e (als)) equivalent to (als) = 0 when (a(a is) = 0 and otherwise Txp.q (a|s) x exp(g(s, a)/r)² The new policy कसत not skewed by the action probabilities in nxp.gi just has the same support.
(D(aas) (1, the above is
THEORETICAL CHARACTERIZATION OF IN-SAMPLE SOFTMAX
In this section we prove in-sample softmax maintains the convergence properties of the standard softmax. In particular, Bellman updates with the in-sample softmax are convergent, and the result- ing in-sample sofumax optimal policy approaches the in-sample optimal policy we reduce the temperature to zero. All proof's are given in Appendix A.
NNot that this softmax operator for the soft Bellman optimality equation is different from the softmax Bellman operator, which
expectation in the hootstrap over softmax policy and which is know have issue with not being an contraction (Asadi Littman, 2017). The log-sum-exp formula is a standard way to approximate the max, and is naturally called softmax.
We define ()
4

---

Published as a conference paper at ICLR 2023
We contrast our in-sample softmax optimality equation in (4) the in-sample Bellman optimality equation introduced by Fujimoto et al. (2019) for the hard max,
(3, = (
+ P(ll,))
max [a' (')))))
Pap (s',
(8)
We first show that the policy produced using the in-sample softmax optimality equation good approximation to that given by the in-sample Bellman optimality equation.
Theorem 1. Let Exp be the in-sample softmax optimal value function. We have lim,-0 Tay = Txp Moreover, let I. be indicator function and xp (als) I[a arg (a) ITD (8,0) a)) ) be the in-sample optimal policy w.r. 9 Define the in-sample softmax optimal policy,
TT log (TD((aa)))
Tx RD (a)s) x (D(aas) exp
We have lim TTD TT'
Now we show that we can reach the in-sample softmax optimal solution, using either value iteration or policy iteration. For value iteration, we define the in-sample softmax optimality operator
(s) P
T llo
edld al'/T
(10)
a':uppa' 8))0
The next result shows that Trp is contraction, and therefore in-sample soft value iteration, using 96+1 = Trp lt is guaranteed to converge to the in-sample softmax optimal value in the tabular case.
Theorem 2. For y 1, the fixed point of the in-sample softmax optimality operator exists and unique. Thus, in-sample soft value iteration converges to the in in-sample softmax optimal value Tep'
As highlighted by Equation (6), the in-sample softmax policy corresponds to the solution of the maximum entropy policy optimization. This implies that similarly to Soft Actor-Critic (Haarnoja et al., 2018), we can apply policy iteration to find this policy. Let Te be the policy at iteration t. The algorithm first learns the value function qmy,, then updates the policy π such that q S 1 The following result shows that this procedure guarantees policy improvement.
Lemma 1. Let Te be a policy such that n. 3 TD. Define
(
T (T(,,0) log To(als)).
T & (D(ass)exp
(1I)
T
Then T++1 I aD and - 2 am
Note that t++1 not only ensures policy improvement, but also stays in the support of TD. Now let us define the on-policy entropy-regularizee operator,
(T"))(s,)) a) r(s. t (1aa))) a') log <a'
(12)
Since this operator is a contraction (shown formally in Lemma 5), evaluate q" by repeatedly applying Thg from any until converge. These updates give rise to the in-sample soft policy iteration algorithm that iteratively updates the policy using (11) and evaluate its by using T The convergence for the tabular case is given below.
Theorem 3. For 7 < 1, starting from any initial policy n such that क I TD, in-sample soft policy iteration converges to the in-sample softmax optimal policy 7
5 POLICY OPTIMIZATION USING THE IN-SAMPLE SOFTMAX
In this section, develop In-sample Actor-critic (AC) algorithm based on the in-sample softmax. This is the first time we see the utility of the in-sample softmax, to facilitate sampling actions from Tp using only actions in the dataset. This contrasts other direct methods, like BCQ that approximate the in-sample max by sampling from an approximate w., (Fujimoto et al., 2019). Throughout this section generically develop the algorithm for continuous and discrete actions. Instead of using
5

---

Published as a conference paper at ICLR 2023
sums, therefore, we primarily write formulas using expectations, which allow for either discrete continuous actions.
The In-sample AC algorithm is similar SAC (Haarnoja al., 2018), except that carefully consider out-of-sample actions. We similarly learn an actor Tæ with parameters v, action-values qo with parameters 0 and value function ve with parameters d. Additionally, we learn au R wp- We need this to define the greedy policy shown above in Equation (11), but do not directly use it to constrain the support over actions.
The first step in the algorithm to extract Tw RJ TD. We do so using simple maximum likelihood loss on the dataset: Lhehavior(4)) (w) - (saaD) [log TT (a]s)]. We do not add any additional tricks to try to ensure action probabilities are where TD(aas) = 0, because this Tu only plays smaller role in our update. will only be used to adjust the greedy policy, and will only be queried on actions in the dataset.
Then we use similar approach to SAC, where we alternate between estimating do and Ui for the current policy and improving the policy by minimizing KI-ddivergence the soft greedy policy. The main difference here to SAC is that we update towards the in-sample soft greedy. We cannot directly use Equation (II), which involves TD the update, but can replace TD in the update with our approximation Tw We therefore update towards an approximate in-sample soft greedy policy
go(s, a) ((s) log TT Tw((s))
aap.qo Qo (a|s) = TD(als) exp
where (s) = = (a]s) exp( gals.a) log Tw (a|s) da is the normalizer to give a valid distri- bution. We minimize a forward KL to this in-sample soft greedy policy, because that allows to sample the KL by only sampling actions from the dataset. To see why, notice that
DKL (+ (+18)| (
= - -Fart (-10) log The (als) - log uTD. (a|s)]
(13)
(0((s,)) a) 2(s) log Th (als)) (log n. (als) + log A (als))
exp
The expectation is now over samples a (-(ss) the actions in the dataset are precisely sampled from TD. To sample the gradient for this loss, we also need an estimate for Z(s). We use our parameterized us estimate Z; we discuss why this reasonable below. The final loss function for the actor T is
(g((s,)) -v((s) log ma(ass) log xp
Luctoe (1b) - EBsanD exp
(14)
For the value function we use standard value function updates for the entropy- regularize setting. The objectives are
1
Ltaseline (Φ) EgwDDanTo (8)
2 (vo(s) (g0(s, a) lo (alss))
(15)
20 (0s a))²
Ceiiii (0) E.
(r 1 roal'')
(16)
The action-values use the estimate of Us in the next state, and avoids using out-of-distribution actions. The update to the value function, Ug. uses only actions sampled from Te. which is being optimized to stay in-sample. Periodically, however, Us may bootstrap off of out-of-distr actions because we do not guarantee that πe 3 TD. In fact, in early learning we expect xp will not satisfy this property. Despite this, the actor update will progressively reduce the probability of these nut-of-distributiin actions, even if temporarily the action-vall values overestimate their value, because the actor update pushes Te towards the in-sample greedy policy. This means that the overestimate is unlikely to significantly skew the actor, and progressively the overestimate should be reduced the support of Te is reduced.
Finally, instead of learning separate approximation for Z, opt for the simpler approach of using Ug. The reason is that Ug should provide a reasonable approximation to because of the relationship between soft values and Z. From Equations (7) and (8) (formally proved in Lemma 3), we know that

---

Published as a conference paper at ICLR 2023
the soft values for the in sampee soft greedy policy Z correspond to the normalizer Z for that policy. Therefore, given that Tu As πD the soft values of approximate in-sample soft greedy policy Top should also be similar to Z. Since optimize policy to approximate T expect its entropy- regulay--e value, which the learning target of up as shown in Equation (15), to be good approximation of Z.
6 EXPERIMENTS
In this section, we investigate three primary questions. First, in the tabular setting, can our algorithm InAC converge policy found by an oracle method that exactly eliminates out-of-distribttion (OOD) actions when bootstrapping? Second, in Mujoco benchmarks, how does our algorithm compare with several baselines using different offline datasets with different coverage? Third, how does InAC compare with other baselines when used for online fine-tuning after offline training? We refer readers to Appendix B for additional details and supplementary experiments.
Baseline algorithms: Oracle-Max: completely eliminates OOD actions when bootstrapping in tabular domains, by using counts exactly estimate Tp. FQI: the regular Q-learning update applied to batch offline data. CQL (Kumar et al., 2020): conservative Q-learning. IQI. (Kostrikov et al., 2022): implicit Q-learning. TD3+BC (Fujimoto & Gu, 2021): TD3 with behavior cloning regularization. AWAC (Nair et al., 2021): Advantage Weighted Actor-Critic.
6.1 SANITY CHECK: APPROACHING ORACLE PERFORMANCE IN THE TABULAR SETTING
In this experiment we demonstrate that InAC finds the same policy as found by an oracle algorithm that completely removes out-of- of--isrributtion (OOD) actions. We use the Four Rooms environment, where the agent starts from the bottom- and needs to navigate through the four rooms to reach the goal in the up-right corner in few steps as possible. There are four actions: A (up, down, right, le ft). The reward zero on each time step until the agent reaches the goal-state where it receives +1. Episodes are terminated after 100 steps, and 7 is 0.9. We use three different behavior policies to collect three datasets from this environment called Expert, Random, and Missing-Action. The Expert dataset contains data collected by the optimal policy. Random dataset, the behavior policy takes each action with equal probability. For the Missing Action dataset, we removed all transitions taking down actions in the upper-left room from the Mixed dataset.
To magnify the impact of bootstrapping from OOD actions we used optimistic initialization for each algorithm (i.e., initialized all action values be larger than the actual values under the optimal policy). This ensures overestimation occurs in some states and we can observe how well the algorithms mitigate poor bootstrap targets.
Export
Random
Micsing-Action Action
80
Oracle-Max
Return
YOL
60
FQI
Episode
per
40
40
40
20
InAC
20
50
75
100
100
25
50
Iteration (x10²)
(a) Policy evaluation performance
(b) Four Room
Figure 1: Policy evaluation performance (return per episode) v.s. number of updates on Expert, Random, and Missing-Action datasets. Each curve is averaged over 10 runs, and shaded areas show a 95% confidence interval.
The results in Figure I are exactly as expected. InAC converges to the same policy as found by Oracle-Max. The FQI baseline cannot effectively remove OOD actions when bootstrapping, and so performs poorly and sometimes completely fails when the dataset has poor action coverage (i.e., there many OOD actions). Finally, IQL performs poorly when the offline data is highly skewed towards suboptimal policies. is likely because the upper expectile of the state value provides a poor approximation to the in-sample maximum action value.
7

---

Published as a conference paper at ICLR 2023
InAC
IQL
TD3+BC
CQL
Ant M-Expert
Ant MiReplay
Hopper M M-Expert
Hopper M M-Keplay
120
120
100
100
90
75
60
50
Normalized Score
30
25
60
80
20
40
60
80
20
40
60
HalfCheetah M-Expert
HalfChhetahh M-Replay
Walker2D M-Expert
Walker2D M-Replay
100
100
75
75
100
75
50
50
50
25
25
25
20
40
60
80
20
40
60
80
0
20
40
60
60
20
40
60
80
Iteration (x10*)
Figure 3: Policy evaluation performance (normalized score) v.s. number of updates. M denotes Medium, The results are averaged over runs, after using a smoothing window of size 10. Results on additional offline datasets are in Appendix B. showing that InAC still learned the best policy.
6.2 OOD EFFECTS IN CONTINOUS ACTION PROBLEMS
In this section we provide a suite of results from four Mujoco environments from D4RL (Fu et al., 2020), now standard datasets for evaluating offline RI algorithms. Fach dataset (named as Ex- pert, Medium-Expert Medium-Replay. and Medium) was designed to mimic different deployment scenarios. the Expert dataset all trajectories were collected using policy learned by a SAC agent. In Medium, all trajectories were collected with the policy learned by SAC agent halfway thought training. Beedium-Exper combines the expert and medium datasets together, and similarly Medium-Replas combines Medium with the replay buffer used during learning.
Figure summarizes each algorithm's performance averaged over all environments under different datasets. Our algorithm's 's performance dominates the others across datasets. In Figure provide more detailed view of the data with learning curves in each environment. Overall InAC performs best or nearly across all domains. In Hopper M-Expert, the result is likely a three-way tie, while in HalfCheetah M-Expert TD3+BC learns faster initially, but the quickly converges to much lower final performance compared with InAC. Naturally, all meth- ods are dependent on the quality the dataset. For example, when shifting from the higher quality (medium-expert) to the lower quality emedium-replay data, 103+BC-whiee regular- izes the policy to stay close to the behavior policy-exhiie a significant performance drop. Overall, TD3+BC and CQL's performance is problem dependent: in some problems perform- ing well and in others basically failing to learn. Finally IQL performs nearly as well as InAC on many problems, but notably not on Walker2D and Hopper M-Replay.
InAC
TD3+ BC
IQL
CQL
120
Normalized Score
90
60
30
Expert
M-Expert
M- M-Replay
Medium
Figure 2: Averaged score over environ- ments v.s. different offline datasets. We averaged the normalized score over four Mujoco tasks and 10 runs. The shaded area indicates 95% confidence inter- val. Comparing InAC and IQL. with the sign test over 40 runs, InAC was sig- nificantly better in all datasets. Expert, M-Expert, and M-Replay had p-value close while Medium dataset gave = 0.002.
These results provide evidence that explicitly avoiding boot- strapping from OOD actions provides significant benefit, but that regularizing the learned policy stay close to the behavior policy can be problematic.
6.3 FROM OFFLINE TRAINING TO ONLINE FINE-TUNING
In real-world applications, it can be useful to take offfine-trained deep RL agent and fine-tune it online. In this section, we investigate how the performance of different baselines changes in fine-tuning, At the beginning of fine-tuning, the agent's policy is initialized with the policy learned offline and the buffer is filled with that same offline dataset. During online interactions, the agent continually adds its new experience into the buffer.
8

---

Published as a conference paper at ICLR 2023
InAC
IQL
AWAC
TD3 +BC
HalfCheetah
Hopper
Walker2D
140
100
120
120
Normalized Score
105
90
90
60
35
25
30
0
M-Expert
M-Replay
M-Expert
M- M-Replay
M-Expert
M- M-Replay
M M-Expert
M M-Replay
Figure 4: Online fine-tuned performance on Medium-Expper and Medium-Replay datasets across four Mujoco environments. M represents Medium in this Figure. The results were averaged over 10 random seeds. The short vertical line indicates the range of 3 times standard error. Each colored bar shows the performance after 0.&M steps of fine-tuning. The thinner black bar inside the colored bar indicates the performance immediately after offline training (i.e., before online fine-tuning). We also report numerical numbers in the Table 13 in Appendix B.1.
Figure 4 shows the policy performance before and after online fine-tuning. We see that InAC is consistently one of the best algorithms across these environments and datasets. There are a few particularly notable outcomes in these experiments. In Hopper and Walker2d for the Medium Medium-Expert data, the performance for IQI. drops significantly after fine-tuning. This contrasts all the other algorithms, which maintained or improved performance when fine-tuning. The cause of this drop is as yet unclear. There is one new algorithm in this set, called AWAC, which was originally proposed specifically for online fine-tuning setting (Nair et al., 2021). We do in-fact see that this algorithm can have quite poor offline performance, but significantly improve after fine-tuning. Despite being designed for this fine-tuning setting, however, it does not outperform the offline algorithms, except in Hopper with Medium- Replay and more minorly on Walker2d with Medium Medium-Replay. Overall, we find that InAC performs well in both the fully offline setting as well as when incorporating fine-tuning.
CONCLUSION
In this paper we considered the problem of learning action-values and corresponding policies from a fixed batch of data. The algorithms designed for this setting need to account for the fact that action-coverage may be partial: certain actions may never be taken in certain regions of the state space. This complicates learning with our algorithms that rely on action-values, estimates q(s, a) and bootstrapping. In particular, if an action a is not visited in s or similar states, the g(s, a) can be an arbitrary value. If this arbitrary value is high, it is likely to be used in the max in the bootstrap target and used to update the policy, which increases probability for high-value actions. This agent is chasing hallucinations, that can produce poor policies even divergence. We focused on a simple approach to mitigate this issue: redefining the objectives to use an in-sample softmax and finally obtaining approach update towards in-sample soft greedy policy that only uses actions sampled from the dataset. The resulting In-sample AC algorithm avoids these hallucinated values when updating the actor, and so correspondingly avoids them when updating the values.
We had two clear findings from this work. First, the move to an in-sample softmax was a key step towards a simple implementation of in-sample learning. Previous work, like BCQ, tried to produce simple algorithm built on an in-sample max algorithm, but needed to incorporate several tricks and later algorithms significantly improve on it. In-sample AC, on the other hand, required only minor modifications to existing AC approaches. The actor update was modified to consider the in-sample softmax, but the resulting update was no more complex than typical actor updates. Second, our results indicate that overall Implicit Q-learning (IQL) is quite good algorithm. Like In-sample AC, also avoids relying on actions sampled from an approximation of TD. but does so using expectile regression. Nonetheless, we find that In-sample AC is always competitive with IQL, and in some cases significantly outperforms it when the dataset is generated by a more suboptimal behavior policy. IQL can still be skewed by too many suboptimal actions in the dataset. In-sample AC provides simple, easy-to-use approach, for both discrete and continuous actions, with an update designed to match only the support of TD and not the action probabilities.

---

Published as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. Proceedings of the 37th International Conference on Machine Learning. 2020.
Kavosh Asadi and Michael Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning, 2017.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- ran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations, 2022.
David Brandfonbrennrr William F Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policc evaluation. In Advances in Neural Information Processing Systems, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Scott Fujimoto and Shixiang Gu. minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, 2019.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, 2021.
Tuomas Haarnjja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2.05905, 2018.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline RL? In International conference on machine learning, 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.
Aviral Kumar, Justin Fu, G. Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 2020.
Xiaoteng Ma, Yigin Yang, Hao Hu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, Bin Liang, and Qihan Liu. Offline reinforcement learning with value-based episodic memory. In International Conference on Learning Representations, 2022.
Jincheng Mei, Chenjun Xiao, Ruitong Huang, Dale Schuurmans, and Martin Müller. On princi- pled entropy exploration in policy optimization. In Proceedings of the 28th International Joint Conference Artificial Intelligence, 2019.
10

---

Published as a conference paper at ICLR 2023
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, 2016.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning, Advances in neural information processing systems, 2017.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. International Conference on Learning Representations, 2021.
Yangchen Pan, Kirby Banman, and Martha White. Fuzzy tiling activations: A simple approach to learning sparse representations online. In International Conference Learning Representations, 2021.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage weighted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint 19xiv:1910.00177, 2020.
Martin I. Puterman. Markav decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Zhao Song, Ron Parr, and Lawrence Carin. Revisiting the softmax Bellman operator: New benefits and new perspective. In International Conference on Machine Learning, 2019.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimiza- tion for offline reinforcement learning. arXiv preprint arXiv:2202.06239, 2022.
Yifan Wu, G. Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, arXiv preprint 1911.::11111110... 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhutdi- nov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In International Conference Machine Learning, 2021.
Chenjun Xiao, Ruitong Huang, Jincheng Mei, Dale Schuurmans, and Martin Müller. Maximum entropy monte-carlo planning. Advances in Neural Information Processing Systems, 2019.
Chenjun Xiao, Yifan Wu, Tor Lattimore, Bo Dai, Jincheng Mei, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. On the optimality of batch policy optimization algorithms. In International conference on machine learning, 2021.
Shentao Yang, Yihao Feng, Shujian Zhang, and Mingyuan Zhou Regularizing model-based policy stationary distribution to stabilize offline reinforcement learning. In International Conference on Machine Learning, 2022.
Yiqin Yang, Xiaoteng Ma. Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 2021.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In Advances in Neural Information Processing Systems, 2021.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. In Conference on Robot Learning, 2020.
Brian Ziebart, Andrew I. Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, 2008
11

---

Published as a conference paper at ICLR 2023
A
APPENDIX: PROOFS
This section includes the proof of all main results.
A.1 RESULTS FOR ONE-STEP DECISION MAKING
We first introduce some results for one-step decision making that will be used in the derivations of main results.
We consider a k-armed one-step decision making problem. Let A be k-dimensional simplex and q (a(1),...,:(()) E IR be the reward vector. Maximum entropy optimization considers
Maximum Entropy Optimization
+ TH(().
(17)
REA
The next result characterizes the solution of this problem (Lemma 4 of (Nachum et al., 2017)).
Lemma 2. For >> 0, let
ea/T
(g = T log Σ eqla)it fr(4) E. esta)/r = g-Fr(a)
(18)
Then there is
(q) max क q + (H((T) = fr(4) q +H((f)))))
(19)
TEA
In-Sample Maximum Entropy Optimization Let B E A be an arbitrary policy. In-sample maximum entropy optimization considers
max π q + +TH(().
(20)
We now characterize the solution of this problem. For > define the in-sample softmax value,
Fg)r log
Σ
ea(a)/t
(21)
a:B(a)>0
and the in-sample softmax policy,
Be9/t-logB (-.
f8) Em:laa)20 es(a)/T Be
-log
(22)
Lemma 3.
Fg.((g) max q TH[((T) x-++TH((T) fa.r(g) 9 -HI(ffarrq))
(23)
Proof. This result is directly implied by Lemma 2.
The next result shows that Fg.r is contractor.
Lemma 4. For any two vectors qı, 92 E R
(F)r(qi)) Fg.r (g2) S [|q] -elloo
(24)
12

---

Published as a conference paper at ICLR 2023
Proof.
(eir -F = sup
{ q1 + +H((T))
sup #2388
{ q2 + +
(25)
30
sup
inf m1 -1 n2
22 + - -
(26)
30
T225
sup {T Q1 π 92
(27)
#38
< max q (a) - (a a:3(a)>0
(28)
S max (a) 92 (a),
(29)
where the first step follows by Lemma 3, the third step follows by choosing T2 = m1. This finishes
the proof.
A.2 RESULT FOR ON-POLICY ENTROPY-RECUILRIIER BACKUP
In this section show some basic results for on-policy most results are generalized from Section C.2 of (Nachum et al., 2017) (which states for e) to
backup. We note that
Recall that the entropy-regularizzed value functions defined
(s) r 7B, (0r (s'),, (*) = E Σ (r((s, log (og
(30)
Define the on-policy entropy-regularized Bellman operator
(T* a) r(s. u) + ("..aa/) =log>(d||<')
log
(31)
Lemma 5, For any policy n, q" satisfies that q" = T Moreover, suppose that |A| < cc. T is
contraction mapping.
Proof. By the definition of ön and q",
(*)) = E Σ (r(s,, - log T T
su =
(32)
r(so,
+
Σ
y (1(8441,0441) - T (agg((++1/ss)))))
80
(33)
==
Bawx(s)
r(s,a) log (als) + YlE
E
? (r(st, - log (at
(34)
Barx(e) [r(s, a) - (a|s) + P( (s"]]]
(35)
Eawx(s)
17th a)) - T Tlo (a|s))
(36)
Thus
7"t (s, a) r(s, + 15 (s')]
(37)
=r(s,0) a) + Eg [Banr(o') (""(s' 1o'g
(38)
=r(s, a) + PT (5','aa') log To'
(39)
=T"Q".
(40)
This finishes the proof of the first part. Since |A| < oc, log (a)ss is bounded for any s, a. Then that Tx is contraction mapping follows directly from standard argument (Puterman, 2014). n
13

---

Published as a conference paper at ICLR 2023
This shows that o is fixed of T". That is, starting from any value q, can learn q" by repeatedly applying q = TTq. The next result characterizes the convergence rate of this algorithm. Lemma 6, For any π and q, we have
9 T0 5 100
(41)
Proof. We prove the result by induction. For the base case, = 0, the result trivially follows. Now suppose that the result holds for k - 1. Then
(I(T)) 4 T-' a0 max (T*)* - (8,
(42)
max
T (T (8, - T (s,
(43)
8,0
y max
P'r [T q (s',
(44)
5,0
T max 8,0
(T*)
q7' (s,a))
(45)
7 ||g q |000
(46)
where the second step uses Lemma 5, the third step uses the definition of T the fourth step uses the Holder's inequality, the last step uses the induction hypothesis. This finishes the proof.
Finally, also need the monotonicity property of the on-policy Bellman operation. Lemma 7. For any n, ifqi (1 92, then T q1 2 92-
Proof. Assume qı 2 q2 and note that for any state-action s, a
( = yEx a' (s) 92(5 0.
(47)
Policy improvement lemma.
Lemma 8. Let be a policy such that π 3 8. Define n²'
q" (s.))) log ((-1s)
<'1ss) B(-ls) exp
(48)
7
Then x' B Bnn gr' 2 9".
Proof. The first part trivially holds by the definition n'.
For the second part, note that by Lemma 3, for any state E S.
n' (+(ss)) (7T (s,:) log # (.1s)) 2 x-118) (s, log n (-1s))
(49)
Then by Lemma 5, for any 8, a E S A,
(" a) = r(s, a) 7 - ( (1,,0)0"" (s', (s',a') a') 7 log π (a' [')ss
(50)
< r(s, a) + 7 o' (1 (T (T's a') - log n' [a|ss')
(51)
<
(52)
< TTh (8,a).
(53)
where we recursively apply Lemma 5 to expand the definition of q" and apply Eq. (49).
14

---

Published as a conference paper at ICLR 2023
A.3 RESULT FOR OFF-POLICY ENTROPY-REGULARIE BACKUP
Given an arbitrary policy 3, consider the following problem
max 1" for all E
(54)
ST 35
For r > 0, define the In-sample softmax Bellman operator
(Tz9)(s,)) = r(s, + 7BBs T log E Ea exp (p((l',
(55)
= =r(s, a) + + (a
(56)
Lemma 9. For y < 1, the fixed point of the in-sample softmax Bellman operator, 9- =TB4' exists and is unique.
Proof. We first show that Tg is contraction. Let aı and d2 be two value functions. Then
[8g 91
T 92
al
max
(1, a) (2)
(57)
8,0
max
Ber (,)) [ (6' ())) Fg. (12(6,))))
(58)
8,0
y max
(s) Fg. (12(s,:))
(59)
S max
(s, (s)
(60)
5.0
7 ll -211
(61)
where the second step uses the definition of Tø the third step uses Holder's inequality, the fourth step uses Lemma
Note that by definition, 93 is the fixed point of Tg. Lemma 10. If q is bounded and q Tá 9. then for any n, q 229999 n"
Proof. We first prove that for any x, q 2 Ts implies that g 2 (7*)*& for & 2 0. Then the result follows by applying Lemma 6. According the assumption,
q Tzq= r(s a) y (g(s)
(62)
a() 7B,
Σ ald'ls') log=(a'l<>))
(63)
=T"g. where the second inequality follows by Lemma 3. Then by Lemma 7,
(64)
2 T 2 T 2 ... 2 (T
(65)
This finishes the proof.
We have the following key result.
Lemma 11. For any E S, 07(s) = max 8* (s).
Proof. We first show 83 2 max ,.. Using the definitions,
0gg Fg.r
(66)
# (Ø) log #3))
(67)
2 π (Ø T looo T
(T 1 9) (a (8)
(68)
2 x ( F log T)
(69)
6*
(70)
15

---

Published as a conference paper at ICLR 2023
where the second and third steps follow by Lemma 3, the fourth step follows by Lemma 10, the last step follows by the definition.
We then prove max D" 2 Uy by first showing that T3 न... Since T3 is the fixed point Tg. by
the uniqueness of the fixed point (Lemma 9), we only need show that T3 g q. This holds because for any as,
Tz (s. = r(s,,) t,Ey
(q
(71)
= r((,, t.y
Σ (s'))
(0"0
log 1ogg)((a'a) #)((a(5'))
(72)
(73)
(s)
(74)
where the second step uses Lemma 3 and the last step uses Lemma Then,
33- (7th T log 33 म (03 log T3) 03
max O
(75)
where the second equality uses that T33 = -0 the last step uses Lemma This finishes the proof.
A.4 PROOF OF THEOREM 1
Theorem (Restatement of Theorem 1). Let l7 be a value function recursively defined as
Ts a) = r(s,a) + BBBowp(lan
Flog
exp (g3(5', (
(76)
a'=)(a'(='>>0
and Rt be a policy defined as
(a(a))) x B(als) exp
log ((als))
(77)
Then we have T3 + 43 and म + 3 as t O.
Proof. By Lemma 11, we have (s, a) max 3 7" (s, a) for any 8, a. The result directly follows by definition of q" The result for policy can be proved similarly.
B APPENDIX FOR EXPERIMENTS
B. ADDITIONAL EXPERIMENTS
This section includes additional experiments to investigate the following questions.
1. We used optimistic initialization for all algorithms in the tabular domain. How do the algorithms perform when using a zero/pessimistic initialization? We show this in Figure 5. In the meanwhile, we added Mixed dataset, which has 1% optimal trajectories and 99% random trajectories.
2. How do our algorithms work on the discrete action domains in the deep learning setting? We show the learning curves Mountain Car, Lunar Lander, Acrobot in Figure 7. The final performance is listed in Figure with normalized score, while the absolute score can be found Figure 9.
3. How do algorithms work more datasets in the continuous action domains? We put learning curves for the expert and medium dataset Figure 10, then list the performance of policies learned with all baselines and all datasets in Figure 11 with normalized score,
16

---

Published as a conference paper at ICLR 2023
while the absolute score can be found in Figure 12. For more Fine-tuning results, put them in Figure 13 with a normalized score, while the absolute score can be found in Figure 14.
4. Will longer run reduce the gap between InAC and baselines? Can InAC still learn better similar policy compared to baselines, if we use another common batch size setting in Mujoco tasks (256)? We changed the batch size to 256 and increased the number of iteration 1.2 million, then show the performance in Figure 15.
5. How does InAC perform in AntMaze? We used antmaze-umaze-- and antmaze-umaze- diverse-vO test InAC, then added the learning curve comparing InAC IQL in Figure 16. We followed the set up in previous work (Kostrikov et al., 2022).
B.2 REPRODUCING DETAILS
This section includes all experimental details to reproduce any empirical results in this paper. We use python version 3.9.6, gym version 0.10.., pytorch version 1.10.0.
B.3 REPRODUCING DETAILS ON TABULAR DOMAIN
Four room environment: The environment is 13 13 gridword, with walls separating the whole space into 4 rooms (as shown in Figure 6). The black area refers to the wall. The agent starts from the lower-left corner and learns to stay in the upper-right comner. When agent runs into the wall, it returns to its previous state. The agent gets a +1 when it transits to the state in the upper-right corner and gets 0 otherwise. The discount rate is 0.9. Thus the upper bound of state value is 10. In tabular experiments, each trajectory was limited to 100 steps. was set to 0.01. A mini-batch update was used. The agent sampled 100 transitions at each iteration. Among the 4 datasets we used, mixed and random datasets used the random restart to ensure full state-action pairs coverage, while expert and missing-action datasets did not.
We used value iterations to find the optimal policy (10k iterations). For the expert dataset, we collect 10k transitions with the optimal policy. For the random dataset, collected 10k transitions with a random restart and equal probability of taking each action. The mixed dataset consists of 100 transitions from the expert dataset and 9900 transitions from the random dataset. The missing action dataset is constructed by all going-down transitions in the upper-left. room from the mixed dataset.
Offline data collection:
Algorithm parameter sweep: The learning rate of InAC, Oracle-Max, and Sarsa had a larger range: The T of in-sample methods was
FQI was swept in [0.1, 0.03, 0.01, 0.003, 0.001]. [0.1, 0.03, 0.01, 0.003, 0.001, 0.0003, 0.0001, 0.00003).
fixed to 0.01.
B.4 REPRODUCING DETAILS OF DEEP RL ALGORIHTM
Network architecture: In mujoco tasks, we used hidden layers with 256 nodes each for all neural networks. In discrete action environments, we used 2 hidden layers with 64 nodes each.
Offline data generation details: In continuous control tasks, we used the datasets provided by D4RL. In discrete control tasks, we used well-trained DQN agent to collect data. The DQN agent had hidden layers with 64 nodes on each, with FTA (Pan al., 2021) activation function on the last hidden layer and ReI.U on others. In Acrobot, the agent was trained for 40k steps with batch size 64. In Lunar Lander and Mountain Car, we trained the agent for 500k and 60k steps separately, with other settings the same as in Acrobot. The expert dataset contains 50k transitions collected with the fixed policy learned by the DQN agent. The mixed dataset has 2k (4%) near-optimal transitions and 48k (96%) transitions collected with randomly initialized policy.
Offline training details: In all tasks, we used minibatch sampling, and the mini-batch size was set to 100. We used the ADAM optimizer and ReLU activation function. The target network is updated by using Polyak average 0.995 x target weight + 0.005 learning weight. We trained the agent for 0.8 million iterations and 70k iterations in mujoco and discrete action environments respectively.
17

---

Published as a conference paper at ICLR 2023
InAC
Oracle-Max
FQI
IQL
Expert
Mixed
Random
Missing Action
80
60
Return per
Episode
20
25
50
75
100
0
25
50
75
100
0
25
50
75
100
0
25
50
75
100
Iteration (x10²)
(a) Initialize weights to 10
Expert
Mixed
Random
Missing Action
80
60
Return
per
40
Episode
20
25
50
100
25
50
75
100
25
75 100
0
25
75
100
Iteration (x10²)
(b) Initialize weights to zero
Expert
Mixed
Random
Missing Action
80
60
Return
per
40
Episode
20
0
25
100
25
100
25
100
25
100
Iteration (x100)
(c) Initialize weights to -20
Figure 5: Learning curves under different initialization
four room gridworld tabular domain.
(a) used 10 to initialize weights, (b) used 0, and (c) used -20. InAC learns reasonable policy with I) expert trajectories, 2) missing action trajectories, 3) mixed trajectories, and 4) random trajectories, where 3) and 4) have full state-action coverage. The results were averaged over 10 random seeds, except that CQL had 5 seeds. The shaded area indicates 95% confidence interval.
Fine Tuning details: We kept all settings as same as in offline learning, and used the learned policy initialization. The offline data was filled into the buffer the beginning of fine-tuning. New interactions would be appended to the buffer later in fine-tuning. No data were removed, The fine-tuning had 0.8M steps.
Policy evaluation details: The policy was evaluated for 5 episodes in the true environment with timeout setting. Acrobot and Lunar Lander had timeout=500, Mountain Car used 2000, and mujoco tasks used 1000. The numbers reported were averaged over 10 random seeds.
Algorithm parameter setting:
18

---

Published as a conference paper at ICLR 2023
InSample
FQI
SARSA
(a) Expert
InSample
FQI
SARSA
(b) Missing Action
Figure 6: Visualization of the learned policy of each algorithms and the estimated values at each state. The blue colors indicate the action value of the corresponding policy and the arrow indicates the action taken by the learned policy. deeper color refers to higher action value. It clear that both FQI and SARSA have serious overestimation and found an incorrect policy when the offline data lacks action coverage (i.e., on the Expert and Missing Missing-action action offline data).
InAC
FQI
SARSA
CQL
IQL
AWAC
Dopart
Hidd
Expen
Hidd
Sapar:
07
10
25
10
(a) Acrobot
(b) Lunar Lander
(c) Mountain Car
Figure 7: Learning curves on the discrete action domains. InAC learns the best policy comparable policy to the strongest baseline. From left to right, we show the result in Acrobot, Lunar Lander, and Mountain Car. In each environment, we tested 2 datasets: Expert and Mixed. In expert dataset, all trajectories were collected with the near-optimal policy, while in mixed dataset, 4% trajectories were optimal and 96% were collected with randomly initialized neural network policy. The y-axis is a normalized return reflecting the performance. The return was normalized according to returns obtained by well trained DQN agent (upper bound) and randomly initialized network (lower bound). The higher the normalized value, the better the performance. The curves were smoothed with window length10. The results were averaged over 10 random seeds. The shaded area indicates 95% confidence interval.
Mujoco tasks: For all algorithms, the learning rate was swept in {3 x 10-4, x 10-4,3 -,3 10-5). InAC swept in (1.0, 0.5, 0.33, 0.1, 0.01). AWAC swept in (1.0, 0.5, 0.33, 0. 0.01). IQL swept expectile in {0.9, 0.7] and temperature in {10.0, 3.0}. The number came from what was reported in the original IQL paper. TD3+BC used a 2.5 in the original paper. CQL-SAC used automatic entropy tuning as in the original paper.
Discrete action enviromments: For all algorithms, the learning rate was swept in (0.003, 0.001, 0.0003, 0.0001, 3e 5, 1e - 5}. For InAC, r was swept in (1.0, 0.5, 0.1, 0.05, 0.01}. IQL. had the same parameter sweeping range as in mujoco tasks. AWAC used À 1.0 in the original paper. CQL used a 5.0.
19

---

Published as a conference paper at ICLR 2023
Acrriie
opt
10110 (9.0000)
(C.005)
0,0997 (0.0061)
1.0040 (0 (0.0011)
1.0077 (0.0015)
1,2059 (0 (0.0077)
1,0214 (0.0007)
moed
0.9970 (8.0000)
0.0051 (0.0030)
0.4041 (0-0153)
0.0834 0004)
(0.0012)
0.8796 (C.0041)
0.8t8l
Lunor Lander mixed
opt
0. 0,2050 (0.0057)
(0.00314)
-0.01.8 (0.0006)
0.2281 (0.0029)
0.3672 (0.0086))
0.1087 (3.0109)
0.0788 (0 0103)
0001)
0.1832 (0 (0.0095)
(6. 00000
0.5832 (9.0014)
0.9849 (2.0017)
0.3084 0103)
(0.0005)
(0.0004)
(0 0293)
(0 0004)
(8.0001)
Mountain Car
mived
0.9853 (8.0008)
0.7229 (0.0352)
0,0130 (0.0085)
0.0890 (3
0.9405 (3,0015)
0.8152 (2,0073)
0.0628 (0.80))
Figure 8: The offinne-trained final performance of each algorithm in discrete action space environ- ments. The number in bracket is the standard error. Scores are normalized. The bold numbers are the best performance in the same setting. Performance was averaged over 10 random seeds.
Enstrommnnt
Densont
WAC
PQI
SARSA
02L
121
AMACC
OraceeMaa
Acrobot
mived
60.02
-136.22 46)
-30.40 20)
.55 (3.41)
.05.4 (0.50)
328.44 69)
-140 140.80 59)
20
-180.09
-188.06 (5
163.71
211.00
101. 001.0001.00 (1.11)
168. (2.2)
ried
(3./5)
-298/3 (8 (8440)
337.84 (7:33)
.98.30 06
-32939 (8.30)
(0.0))
Mountii
epe
-108 14 (3
-51..66 (2.35)
-1988.22 76 (0
-276 90 (8.
-118.00 (0.05)
-110.08 (0.22)
-175.90 (1.
Figure 9: The offline-trained final performance of each algorithm in discrete action space environ- ments. This table reports the return per episode before normalization. The number in bracket is the standard error. The bold numbers are the best performance in the same setting. Performance was averaged over 10 random seeds.
InAC
CQL
Ant Expert
Medium
Hopper Expert
Hopper Medium
100
100
90
90
Normalizes Ized Score
30
30
25
25
20
60
60
40
40
HalfCheetah Expert
HalfCheetah Medium
Walker2D Expert
Walker2D Medium
100
100
100
50
50
50
50
25
25
0
0
20
60
80
20
40
60
40
80
20
40
80
Iteration (x10")
Figure 10: Learning curve on the mujoco tasks. InAC learns the best policy or a comparable policy the strongest baseline. The results averaged over 10 random seeds, except that CQL had seeds. The shaded area indicates 95% confidence interval.
20

---

Published as a conference paper at ICLR 2023
Environment
Dataset
InAC
IQL
AWAC
TD3+BC
CQL
expert
128.40 (0.42)
118.80 (0.50)
100.60 (0.89)
93.30 (1.04)
86.00 (1.50)
medium
120.90 (0.56)
121.00 (0.58)
97.80
68.70 (0.89)
111.40 (0.86)
Art
medium replay
(0.63)
(0.44)
59.20 (0.39)
(0.84)
94.20 (0.88)
94.50 (0.49)
84.10 (0.30)
89.80 (0.r3)
98.20 (0.10)
expert
93.60 (0.04)
91.50 (0.20)
10.00 (0.10)
92:10 (0.03)
5.20 (0.17)
medium expert
83:50 (0.34)
83:40 (0.46)
38.50 (0.30)
54.10 (0.38)
22 20 (0.12)
HalfCheetah
medium replay
44.30 (0.02)
45.00 (0.03)
44, 44.80 (0.01)
35.00 (0.20)
46.60 (0.02)
medium
48.30 (0.02)
48.50 (0.02)
37
32.10 (0.36)
47.60 (0.02)
expert
(0.38)
(0.74)
(0.97)
78.90 (0.91)
(0.61)
expert
(0.68)
(0.95)
(0.34)
(0.26)
(1.10)
Hopper
medium replay
92.10 (0.38)
60.30 (0.44)
71.20 (0.97)
20.30 (0.19)
71.20 (0.92)
medium
60:30 (0.20)
59.00 (0.26)
72.70 (0.45)
50.20 (0.13)
69.80 (0.47)
expert
110.60 (0.09)
102.80 90 (0.60)
110.20 (0:03)
79.40 (1.04)
106.00 (0.35)
medium expert
109.00 (0.10)
(0.35)
64)
(0.25)
Walker2D
medium replay
(0.57)
(0.53)
(0.32)
(0.35)
(0.18)
medium
82.70 (0.16) 71.10 (0.53)
13.20 (0.87)
54.10 (0.65)
82.40 (0.06)
Figure 11: The final performance in continuous action space environments. The number in bracket is the standard error. Scores are normalized. The bold numbers are the best performance in the same setting. Performance was averaged over 10 random seeds, except that CQL had 5 seeds.
Environment
Datasel
InAC
IQL
AWAC
TD3+BC
CQL
expert
5077.97 (17.92)
4685. (20.71)
3911.03 (37.60 60)
3593.61 (43.41)
3288.60 (63.46)
medium expert
4763.72 (24.29)
3767.70 (65.10)
(37.71)
(37.01)
Ant
replay
(26.84)
3427.12 (18.50)
(48.23)
(16.45)
3259.72 (31.60)
medium
3637.89 (36.90)
3642.84 (20.60)
3211.47 (12.68)
2606.69 (29.86)
3846.52 (4.00)
expert
11347.06 (5.12)
11089.65 (24.91)
959.41 (12.87)
11151.42 (4.46)
354.04 (21.25)
medium expert
10086.43 (42.11)
10054.48 46 (56.34)
4511 98 (36 84)
6431. 29 (47.99)
2485. 94 (13.24)
HallCheetai
medium replay
5209.67 (2.66)
5269.54 (3.22)
5275.87 (1.27)
4096.27 (24.64)
5495.89 (1.68)
medium
(9)
(2.02)
4359.90 (52.57)
(44.49)
(3.18)
expert
3346.42 (12.48)
2885. (24.07)
1445 60 (31.53) 53)
2544..9 (29.500 (99.50)
3053. (19.96)
medium expert
3032.05 (22.29)
1982.46 (31.03)
733.59 (10.98)
1635.68 (9.51)
2504.53 (36.45)
Hopper
medium replay
2975.21 (12.04)
1942.17 (15, (5)
2294.38 (31.45)
638. 109.10066 (6.33)
2501 46 (30.08)
medium
1945.62 (6.19)
1897.93 (9
2340.96 (14.89)
1609 48 (4.22)
2252.67 (15.40)
expert
5076.38 (4.04)
4725.18 (27.57)
5061.99 (1.02)
3644.08 09
4868.31
expest
5006.62 (4.27)
4410.62 (166433)
2361.18 (60.66) 66)
(29.44)
4804.33 (11.52)
Walker2D
medium replay
3205.26 (26.23)
3034.25 (24.45)
3553.79 (14.45)
769.40 (15.84)
3698.78 (8.32)
medium
3790.83 (7.16)
3269 B4 (24.45)
600.84 (39.67)
2474.75 (29.75)
3778.22 (2.69)
The absolute final performance continuous action space environments. This table reports the score before normalization. The number in bracket is the standard error. The bold numbers are the best performance in the same setting. Performance was averaged over 10 random seeds, except that COL had 5 seeds.
Figure 12:
21

---

Published as a conference paper at ICLR 2023
Envrronment
Dataset
Pertormance After
InAC
IQL
AWAC
TD3:BC
Offline
128.40 (0.42)
118.80 (0.50)
100. 60 (0.89)
93.30 (1.04)
expert
FineTune
134.80 (0.11)
112.90 (0.69)
134.70 (0.26)
116.30 (0.73)
Change
6.4
-5.9
34.1
23
Offine
120.90 (0.58)
121.00 (0.58)
97.80 (1:55)
88.70 (0.89)
medium
FineTune
132.60 (0.46)
117.20 (1.03)
102.50 (2.80)
117.00 (0.27)
Change
Ant
Offine
(0.63)
(0.44)
(1.15)
(0.39)
replay
(0.10)
(0.36)
(2.68)
(0.41)
Change
26.8
20.1
-15.6
12.2
Offiine
94.20 (0.88)
94.50 (0.48)
84.10 (0.30)
69.80 (0.71)
medium
FineTune
122.50 (0.04)
109.20 (0.30)
-42.20 (0.25)
82.30 (0.53)
Change
28.3
14.7
-126.3
12.5
Offine
93.60 (0.04)
91.50 (0.20)
10.00 (0.10)
92.10 (0.03)
experi
FineTune
92.40 (0.08)
94.00 (0.04)
54.80 (0.27)
90.90 (0.03)
Offine
(0.34)
(0.30)
(0.38)
expert
(0.05)
(0.05)
(0.17)
(0.32)
HallChestah
Change
9.4
9.2
47.1
-3.3
Offine
44.30 (0.02)
45.00 (0.03)
44.80 (0.01)
35.00 (0.20)
medium replay
FineTune
55.60 (0.13)
51.70 (0.10)
55.60 (0.08)
34.70 (0.12)
Change
11.3
6.7
10.8
-0.3
Offine
48.30 (0.02)
48.50
37.40 (0.43)
32.10 (0.35)
medium
FineTune
63.80 (0.07)
54.80
62.20 (0.09)
42.60 (0.02)
Change
Offine
(0.30)
(0.74)
(0.97)
(0.91)
expert
FineTune
(0.17)
67.40 (1.02)
77.90 (1.17)
91.40 (0.52)
Change
-22
32.9
12.5
Offine
93.80 (0.89)
61.80 (0.95)
23.30 (0.34)
50.80 (0.26)
medium expert
FineTune
93.30 (1.03)
49.00 (1.48)
89.10 (0.83)
54.90 (0.23)
Change
-0.5
-12.8
65.8
4.1
Hopper
Offine
92.10 (0.38)
60.30 (0.49)
71.20 (0.97)
20.30 (0.19)
medium replay
FineTune
97.80 (0.17)
61.10 (1.33)
102.60
25.50 (0.26)
Change
(0.20)
(0.26)
(0.45)
(0.13)
Fine
(0.35)
(0.61)
(0.27)
(0.15)
Change
3.4
21.9
0.9
Offiine
110.60 (0.06)
102:90 (0.60)
110.20 (0.03)
79:40 (1.04)
expert
FineTune
110.90 (0.03)
105.50 (0.21)
107.40 (0.44)
108.70 (0.03)
Change
0.3
2.6
-2.8
29.3
Offine
109.00 (0.10)
96.00 (0.35)
51.20 (1.32)
89.70 (0.64)
medium
FineTune
112.20 (0.03)
44.20 (0.46)
64.30 (1.48)
93.90 (0.54)
Change
-51.8
33.1
4.2
WalkerDD
Offiie
(0.57)
(0.53)
(0.32)
(0.35)
replay
(0.10)
(0.51)
(0.10)
(0.23)
Change
25.7
19.5
19.6
-3.3
Offine
62.70 (0.16)
71.10 (0.53)
13.20 (0.87)
54.10 (0.65)
mediuum
FineTune
89.70 (0.07)
79.50 (0.26)
10.10 (0.98)
66.40 (0.27)
Change
8.4
-3.1
12.3
Figure 13: The performance changes during fine-tuning. The number in bracket is the standard error. Scores normalized. Performance averaged over 10 random seeds.
22

---

Published as a conference paper at ICLR 2023
Environment
Dataset
Perfonmance After
InAC
IQL
AWAC
TD3+BC
Offiine
(17.92)
(20.71)
3593.61 (43.41)
expert
(4.65)
4423.00 (29.06)
(10.96)
4573.23 (30.96)
Change
Offine
4750.21 (24.2)
4763.72 (24.29)
3787. 70 (65.10)
3403.58 (37.71)
medium expert
FineTune
5254.04 (19.83)
4539.00 0399.00 (43.19)
39833.4 (117.36)
4506.45 (11.26)
Change
503.83
-164.72
196.72
1192.91
Offine
3391.06 (26.64)
3427 2.27.2 (18.50)
2410.17 (48.23)
2161.59 (16.45)
medium replay
FineTune
4512.61 (4.01)
4269.35 (15.13)
1752 (112.73)
2573.63 (17.31)
Change
1121.54
842.24
657.3
512.04
Offiine
3637 (36.50
3642 (20.60)
3211.47 (12.66)
2606.69 (29.86)
(1.55)
4286.09 (12.80)
(10.54)
(22.55)
Change
623.25
-5315.46
525.43
Offine
11347.06 (5.12)
11009.65 (24.91)
969.41 (12.87)
11151.42 (4.46)
experi
FineTune
11196.82 (9.57)
11408.80 (4.98)
6546.81 (33.36)
11010.34 (4.78)
Change
-150.24
319.15
5587.4
-141.08
Offine
10096.43 (42.11)
10054.46 46654446666 (56.34)
4511.98 (36.84)
6431.29 (47.99)
medium
FineTune
11262.88 (6.22)
11218.53 (5.95)
10347.9 19347.94 (20.20)
6034.64 (40.49)
Change
1176.45
1164.07
5835.96
-396.65
HalfChestah
Offiine
5209.87 (2.68)
5289 (3.22)
5275.87 (1.27)
4096.27
replay
(16.13)
6156.15 (12.87)
(9.20)
(15.08)
Change
1425.45
1.352.48
-74
Offiine
5716.72 (2.29)
5742. 89 (2.02)
4358. 90 (52.57)
3696. 49 (44.49)
medium
FineTune
7639.17 (8.72)
6513.38 (16.75)
7439:24 (11.91)
5005.84 (2.67)
Change
1922.45
770.49
3079.34
1309:35
Offine
3346.42 (12.48)
2885.90 90 (24.07)
1445. 60 (31.53)
2544.39 (29.50)
expert
FineTune
3517.17 (5.66)
2168.89 (33.35)
2515.50 (77.66)
2955.89 (16.71)
Change
170.75
-717.02
1069.9
411.61
Offiine
29)
1992 (31.03)
(10 99)
1635.83 (8.51)
medium expert
(33.73)
(48.16)
2878.88 (27.2))
1769.68 (7.49)
Change
133.85
Hopper
Offine
2975.21 (12.04)
1942.17 (15.85)
2294.38 (31.45)
638.10 (8.33)
medium replay
FineTune
3160.03 (5.50)
1985.99 (43.32)
3318.32 (2.89)
806.58 (8.47)
Change
184.82
23.82
1023.93
168.47
Offine
1945.62 (6.19)
1897.93 (9.11)
2340.96 (14.89)
1609.48 (4.22)
medium
FineTune
2593.49 (11.20)
2006.70 (20.02)
3064.90 (8.81)
1637.37 (4.88)
Change
647
108.76
723.94
27.89
Offinee
5076.38 (4.04)
4725.18 (27.57)
5061.92 (1.02)
3844.09 (47.59)
expert
(1.54)
(9.34)
(20.22)
(1.37)
Change
1340.04
Offine
5006.62 (4.27)
4410.62 (16.43)
2361.18 (60.66)
4120.01 (29:44)
medium expert
FineTune
5149.60 (1.53)
2036.23 (21.97)
3650.85 (67.75)
4314.14 (24.81)
Change
142.98
-2374.39
1497.67
194.13
WalkerZD
Offine
3206. 26 (26.23)
3034.26 25 (24.45)
3562.79 (14.45)
789.40 (15.84)
medium replay
FineTune
4391,99 (4.7 (4.74)
3937. 08 (23.09)
4453.95 (4.48)
634.43 (10.68)
Change
11.88.7
902.83
900.16
-154.97
Offiiee
3790.83 (7.15)
3289 (24.45)
800.84
2474.75 (29.75)
medium
(3.28)
3650.39 (12.29)
(45.64)
(12.63)
Figure 14: The performance changes during fine-tuning, This table reports the score before normal- ization. The number in bracket is the standard error. Performance was averaged over 10 random seeds.
23

---

Published as a conference paper at ICLR 2023
InAC
10L
AWAC
TD3-BC
COL
Ant Expert
Ant M Expert
Ant Replay
Ant Medium
90
90
90
90
60
60
60
60
30
30
30
40
80
120
40
80
120
40
80
120
40
80
120
HalfCheetah Expert
HalfCheetah M-Expert
HalfCheetah M-Replay
HalfCheetah Medium
100
100
100
100
75
75
50
50
50
25
25
25
0
40
80
120
40
80
120
40
80
120
40
120
Normalizeed Score
Hopper Expert
Hopper M-Expert
Hopper M-Replay
Hopper Medium
100
100
100
100
75
50
50
50
25
25
25
25
40
80
120
40
80
120
40
80
120
40
120
Walker2D M-Expert
ExExert
M-Replay
Medium
100
100
100
100
75
75
50
50
50
25
25
25
25
40
80
120
80
120
120
Iteration (x104)
Figure 15: Offline learning curves with 1.2 million iterations. The x-axis is the number of iterations and the y-axis is the normalized score. Performance averaged over 5 random seeds, after using smoothing window of size 10.
InAC
IQL
Normalized Score
Umaze
Umaze-Diverse
100
100
75
75
25
25
0
25
50
100
25
75 100
Iteration (x10*)
Figure 16: Offline learning curves with million iterations. The x-axis is the number of iterations and the y-axis is the normalized score. Performance was averaged over 5 random seeds, after using smoothing window size 10.
24


Reviews:
[{"review_id": "emtE4zrvQ7", "reviewer": "Reviewer_Dopc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In maximum entropy RL, we use a soft Bellman optimality equations where the standard max operation is replaced by the log-sum-exp operation. Just as researchers have proposed to use in-sample max instead of max over the entire action space, this paper propose to use in-sample softmax to improve the softmax operation, such that we only uses actions well-covered by the dataset. Some theoretical study are provided to show the convergence property similar to softmax. Experimental results demonstrate the effectiveness of the proposed in-sample softmax operation for a SAC-type algorithm on various offline RL benchmarks.", "summary_of_the_review": "The paper propose to use in-sample softmax instead of softmax for maximum entropy offline RL, which leads to in-sample SAC that shows promising results on some offline RL benchmarks. The idea is interesting, though being a straightforward extension of existing ideas.", "strength_and_weaknesses": "Strength:\n- The paper is well-written and easy to follow.\n- Nice theoretical characterization.\n- Experiments results seem promising.\n\nWeaknesses/Questions:\n- The idea is a bit straightforward (though not necessarily a weakness), as we have in-sample max to improve max, it seems not surprising to use in-sample softmax to replace softmax.\n- The explanation/derivation in page 4 before section 4 may not be necessary as the math seems trivial. Also some argument could be more accurate (e.g. when $\\pi_D(a|s) = 0$, equation (7) is undefined since $\\log(\\pi_D(a|s) )$ does not exist).\n- The in-sample softmax optimal policy approaches the in-sample optimal policy as the temperature approaches zero. In practice, how do you choose the temperature. Have you considered annealing the temperature for better performance?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F3xuWMPVLc", "reviewer": "Reviewer_xpgy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an Actor-Critic RL algorithm able to learn from an offline dataset, while addressing the problem of out-of-distribution actions. The core contribution of the paper is the identification of how nicely soft policies and Q-updates (based on the softmax over actions) are amenable to updates robust to out-of-distribution actions. The resulting algorithm looks like the Soft Actor-Critic, but with the actor and critic losses slightly modified to introduce a term that depends on the estimated behavior policy used to collect samples in the dataset. Experimental results show that the resulting algorithm outperforms several baselines on standard benchmarks, and a theoretical analysis explains why the algorithm makes sense.", "summary_of_the_review": "Very well-written paper with sufficient (but can be improved) empirical results. I would strongly advocate for the authors to include raw (un-normalized) scores in the final version of the paper, for ease of comparison against algorithms not present in this particular paper.", "strength_and_weaknesses": "Strengths:\n\n- The paper is very interesting to read and the idea is exciting! Some parts of the paper, that may appear simple (such as Equation 5, the trick to reduce the important of $\\pi_D$), are eye-opening.\n- The proposed algorithm is a relatively simple modification of the standard soft actor-critic losses. Nothing extra to compute, no resampling of the dataset to do. The algorithm seems easy to implement, and at almost no computational cost compared to vanilla SAC.\n- The paper is very clear and well-presented. Just the right amount of maths is given to understand how the new losses are designed, why they make sense, and how they can be implemented. Equations 14 to 16 are nicely grouped together to give a direct overview of the full (modified) SAC algorithm.\n- The sanity check is very nice to have, more papers should have a small and simple sanity check like this.\n\nWeaknesses:\n\n- The empirical evaluation is not as precise and complete as in other papers. Even in the appendix, I did not find absolute scores obtained by the agents in the environments (to be easy able to verify that the baseline scores match what other papers report, and to easily compare the scores obtained by IAC with baselines not present in the paper). Normalized scores are not that useful and usually hide more information than they reveal. Evaluation on AntMaze could also have been interesting: some people motivate that AntMaze is more difficult than the Mujoco tasks due to its sparse reward function, that better stresses the ability of an RL agent to bootstrap (instead of just following $r_t$).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EMeC-PPCMGS", "reviewer": "Reviewer_LBrh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an offline RL algorithm based on the principles of in-sample maximization (when fitting the Q function) and maximum-entropy RL. It introduces the in-sample softmax, which is a straightforward modification of the usual softmax used in MaxEnt RL to only include actions in the support of the behavior policy, similarly to Batch-Constrained Q-Learning (BCQ). The convergence of in-sample soft policy iteration to an optimal policy (for an appropriate definition of optimal) is proved.\n\nThe main novelty is in showing that this in-sample softmax can be written as an expectation over samples from the dataset, reducing dependence on the behavior policy. Using this identity, the authors develop an algorithm similar to Soft Actor-Critic (SAC), which they dub In-sample Actor-Critic (InAC). Experiments demonstrate that InAC is effective in avoiding out-of-distribution (OOD) actions and attains high performance in both purely offline and offline-to-online settings.", "summary_of_the_review": "I think the paper presents a useful idea and algorithm with compelling results, and therefore should be accepted.", "strength_and_weaknesses": "Strengths:\n* The paper presents a clean motivating theory that is not too far removed from the practical implementation.\n* Unlike some other offline RL algorithms, InAC does not enforce constraints between the learned policy and the estimated behavior policy. This is good because imperfections in the estimated behavior policy can bias learning.\n* Experiments show that the algorithm can effectively avoid OOD actions.\n* The performance of the algorithm is similar to or better than existing algorithms, both for offline and offline-to-online learning.\n\nWeaknesses:\n* Judging by Figure 3, it seems that some of the algorithms have not converged within the 0.8M iterations. Obviously it is good that InAC learns faster, but it’s possible that the other algorithms could reduce the gap if given more iterations. Since training time is typically not the constraining factor in offline RL, I think all algorithms should be run to convergence.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%