In-sample Actor Critic for Offline Reinforcement Learning
Authors: Hongchang Zhang
Yixiu Mao
Boyuan Wang
Shuncheng He
Yi Xu
Xiangyang Ji
Venue: ICLR 2023 poster (2023)
Keywords: offline reinforcement learning

Abstract: Offline reinforcement learning suffers from out-of-distribution issue and extrapolation error. Most methods penalize the out-of-distribution state-action pairs or regularize the trained policy towards the behavior policy but cannot guarantee to get rid of  extrapolation  error. We propose In-sample Actor Critic (IAC) which utilizes sampling-importance resampling to execute in-sample policy evaluation. IAC only uses the target Q-values of the actions in the dataset to evaluate the trained policy, thus avoiding extrapolation  error. The proposed method performs unbiased policy evaluation and has a lower variance than importance sampling in many cases. Empirical results show that IAC obtains competitive performance compared to the state-of-the-art methods on Gym-MuJoCo locomotion domains and much more challenging AntMaze domains.

Published as a conference paper at ICLR 2023
SN-PE ACTOR CRITIC FOR OFFLINE REINFORCE- MENT LEARNING
Hongchang Zhang", Yixiu Mao" ,Boyuan Wang', SShmcceng He', Yi Xu², -aee,,Xannnnggg Ji
Tsinghua University Daalin University of Technology
(hc-zhang1 9, myx21, wangby22, hescl 6) @ma i 1s. wts.tsinngg tww.gguanaa..... edu, yxu@dlute edu,, xyji@tsinghua.
ABSTRACT
Offline reinforcement learning suffers from out-of-distribution issue and extrap- olation error. Most methods penalize the out-of-distribution state-action pairs regularize the trained policy towards the behavior policy but cannot guarantee to get rid of extrapolation emror. We propose In-sample Actor Critic (IAC), which importance resampling to execute in-sample policy evaluation. IAC only uses the target Q-values of the actions in the dataset evaluate the trained policy, thus avoiding extrapolation error. The proposed method performs unbiased policy evaluation and has lower variance than importance sampling in many cases. Empirical results show that IAC obtains competitive performance compared to the state-of-the-a methods on Gym-MuJoCo locomotion domains and much more challenging AntMaze domains.
utilizes
INTRODUCTION
Reinforcement learning (RL) aims to solve sequential decision problems and has received extensive attention in recent years (Mnih et al., 2015). However, the practical applications of RL meet several challenges, such risky attempts during exploration and time-consuming data collecting phase. Offline RL capable of tackling these issues without interaction with the environment. It can get rid of unsafe exploration and could tap into existing large-scale datasets (Gulcehre et al. 2020
However, offline RL suffers from out-of-distribution (OOD) issue and extrapolation error (Fuji- moto al. 2019). Numerous works have been proposed to overcome these issues. One branch of popular methods penalizes the OOD state-action pairs or regularizes the trained policy towards the behavior policy (Fujimoto & Gu 2021: Kumar et al. 2020). These methods have to control the degree of regularization to balance pessimism and generalization, and thus are sensitive to the regularization level (Fujimoto & Gu (021). In addition, OOD constraints cannot guarantee avoid extrapolation error (Kostrikov et al. 2022). Another branch chooses to eliminate extrapolation error completely (Brandfonbrener et al.) 2021] Kostrikov al., 2022). These methods conduct in sample learning by only querying the Q-values of the actions in the dataset when formulating the Bellman target. However, OneStep RI. (Brandfoobrrnner et al., (02T) estimates the behavior policy's Q-value according to SARSA (Sutton & Barto (0088) 8) and only improves the policy a step based the Q- value function, which has a limited potential to discover the optimal policy hidden in the dataset. IQL (Kostrikov et al. (022)) relies on expectile regression to perform implicit value iteration. It can be regarded as in-support Q-learning when the expectile approaches 1, but suffers from instability in this case. Thus suboptimal solution is obtained by using smaller expectile. Besides, these two lines of study adapt the trained policy to the fixed dataset's distribution.
Then question appears- "Can introduce the concept of in in-sample learning to iterative policy iteration, which is a commonly used paradigm to solve RI.? General policy iteration cannot be updated in in-sample style, since the trained policy will inevitably produce actions that of the dataset (out-of-sample) and provide overestimated Q-target for policy evaluation. To enable in-sample learning, we first consider sampling the target action from the dataset and reweighting
Equal contribution.

---

Published as a conference paper at ICLR 2023
the temporal difference gradient via importance sampling. However, it is known that importance sampling suffers from high variance (Precup et al. 2001) and would impair the training process.
In this paper, propose in-sample Actor Critic (LAC), which performs iterative policy iteration and simultancously follows the principle of in-samlle learning to eliminate extrapolation error. We resort to sampling-importe importance resampling (Rubin] 1988) reduce variance and execute in-sample policy evaluation, which formulates the gradient as it is sampled from the trained policy. To this end, we use SumTree to sample according to the importance resampling weight. For policy improvement, we tap into divantage-weighted regression (Peng et al., 2019) to control the deviation from the behavior policy. The proposed method executes unbiased policy evaluation and has smaller variance than importance sampling in many cases. We point out that, unlike previous methods. IAC adapts the dataset's s distribution to match the trained policy during learning dynamically. We test IAC on D4RL benchmark (Fu et al., 2020), including Gym-Mulod locomotion domains and much more challenging AntMaze domains. The empirical results show the effectiveness of IAC.
2 RELATED WORKS
Offiine RL. Offline RI., previously termed batch RL (Ernst et al. 2005 Riedmiller 2005), pro- vides a static dataset to learn policy. It has received attention recently due to the extensive usage of deep function approximators and the availability of large-scale datasets (Fujimoto et al. 2019
Ghasemipour et al. 2021). However, it suffers from extrapolation error due to OOD actions. Some works attempt to penalize the Q-values of OOD actions (Kumar et al. 2020 An et al. 202T). Other methods force the trained policy be close to the behavior policy by KL divergence (Wu et al.
2019), behavior cloning (Fujimoto & Gu 2021), or Maximum Mean Discrepancy(MMD)) (Kumar al. 2019). These methods cannot eliminate extrapolation error and require a regularization hyper- parameter to control the constraint level to balance pessimism and generalization. Another branch chooses to only refer to the Q-values of in-sample actions when formulating the Bellman target without querying the values of actions not contained in the dataset (Brandfonbrener et al. [202]] Kostrikov et al. 2022). By doing so, they can avoid extrapolation error. OneStep RL (Brandfon- brener et al. (022)) evaluates the behavior policy's Q-value function and only conducts one-step of policy improvement without off-policy evaluation. However, it performs worse than the multi-step counterparts when large dataset with good coverage is provided. IQL (Kostrikov et al. (0022) draws on expectile regression to approximate an upper expectile of the value distribution, and executes multi-step dynamic programming update. When the expectile approaches 1, it resembles in-support Q-learning in theory but suffers from instability in practice. Thus a suboptimal solution is obtained by using smaller expectile. Our proposed method opens up a venue for in-sample iterative policy iteration. It prevents querying unseen actions and is unbiased for policy evaluation. In practice, modifies the sampling distribution to allow better computational efficiency. OptiDICE (Lee al., 2021] also does not refer to out-of-sample samples. However, it involves with complex minmax optimization and requires a normalization constraint to stabilize the learning process.
Importance sampling. Importance sampling's application in RL has a long history (Precup 2000 for its unbiasedness and consistency (Kahn & Marshall (953). Importance sampling suffers from high variance, especially for long horizon tasks and high dimensional spaces (Levine et al. 2020).
Weighted importance sampling (Mahmood et al. 2014, Munos et al,, (016) and truncated impor- tance sampling (Espeholt et al., 2018) have been developed to reduce variance. Recently, marginal- ized importance sampling has been proposed to mitigate the high variance of the multiplication of importance ratios for off-policy evaluation (Nachum et al.) 2019 Liu et al. 2018). Sampling- importance resampling an alternative strategy that samples the data from the dataset according to the importance ratio (Rubin 1988 Smith & Gelfand 1992 Gordon et al., 1993). It has been
applicd in Sequential Monte Carlo sampling (Skare et al. 2003) and off-policy evaluation (Schlegel eet al. 2019). To the best of our knowledge, our work is the first to draw sampling-importane resampling to solve the extrapolation error problem in offline RL.
PRELIMINARIES
RL. In RL, the environment is typically assumed to be a Markov decision process (MDP) (S, A, R,),7), with state space S, action space A, scalar reward function R. transition dynam-
2

---

Published as a conference paper at ICLR 2023
ics p. and discount factor (Sutton & Barto 2018). The agent interacts with the MDP according to a policy Z s). which is a mapping from states to actions (deterministic policy) or a probabil- ity distribution over actions (stochastic policy). The goal of the agent is to obtain policy that maximizes the expected discounted return: E,, - 7'r.]. Off-policy RL methods based on ap- proximate dynamic programming typically utilize state-action value function (Q-value function), which measures the expected discounted return obtained by starting from the state-action pair (s, a)
and then following the policy क: Q(s, a) Ex, [2r=0 7'r, 80 8, 0g = a)
Offiine RI. In offline RI,, the agent is prohibited from interacting with the environment. Instead, is provided with fixed dataset collected by some unknown behavior policy 33. Ordinary approx- imate dynamic programming methods evaluate policy ı by minimizing temporal difference error, according to the following loss
(f) = [(sas))= a) (1)) Qú(s' a') Q))
where D is the dataset, T is policy parameterized by 6, Qs,s, a) is Q function parameterized by 0, and Qg(s, a) is a target network whose parameters are updated via Polyak averaging.
We denote the 1th transition (s,, a Fr, s/ a²) in D as 24, and D = {x1,... , J. For some transition (s, r,, d'), let transition-wise TD update A (ar) be the gradient of ransition-wiss TD error,
(x) = VoQo (Qo((s a) - r(s, a) - (''''a')))
For the convenience of subsequent theoretical analysis, we also define the expected value of the TD update based on the gradient of Equ. (I by replacing the empirical distribution of the dataset with induced distribution
ATD = (Ate-,-((())))
(2)
(dd())))) where Px = P(sls (a'ls"s and (s) is the normalized and dis- counted state-action occupancy measure of the policy B. That is, d (s. = (1
7'1 ar a) (. ss)].
Besides policy evaluation, a typical policy iteration also includes policy improvement. In continuous action space, stochastic policy can be updated by reparameterizzation:
+ argmaX, E.-DD (Qu (s, fo (c; s))l
(3)
where N a Gaussian distribution. In offline RL, OOD actions a' can produce erroneous values for (s s a') in Q-value evaluation and lead to inaccurate estimation of Qavuue. Then in policy improvement stage, where the policy is optimized to maximize the estimated Qp. the policy will prefer OOD actions whose values have been overestimated, resulting in poor performance. Most current methods either directly constrain policy π or regularize OOD actions' values to constrain policy indirectly.
IN-SAMPLE ACTOR CRITIC
In the following, introduce allow in-sample learning. Then we show that the proposed method is unbiased and has a smaller variance than importance sampling in many cases. Last, we present a practical implementation of the algorithm.
resampling and tap into it for offline RL to
4.1 SAMPLING-IMPORTANCE RESAMPLING
Consider a statistical problem relevant to offline RL. Suppose we want to obtain samples from a target distribution p(z). but we only have access to samples from some proposal distribution q(x)). How can we simulate random draws from p? Assuming supp(g) 2 supp(p), classic algorithm importance resampling (SIR) (Rubin 1988 addresses this with the following procedure:
Step 1.(Sampling) Draw independent random samples (#1...., In from q.
Step 2. (Importance) Calculate the importance ratio for each æ;: w(x,) p(x.)/a(ax)
We omit the learning rate for simplicity
3

---

Published as a conference paper at ICLR 2023
U[ MOO,
A1 0.5) to XX0055
Start
End
Start
End
0.8
0.8-
0.8
0.8
density
0.6
sample deisity
0.6
density
0..6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.0
-1
0.0
0.0
0.0
0
X
Figure 1: SIR for distribution correcting, We generate 100,000 random values from proposal distri- bution 4, and use SIR to resample 10,000 random items out of to approximate target distribution Sample histograms and the actual underlying density of both g(start) and p(end) are presented. Uniform ( to Gaussian N (100, 0.5). Right: Gaussian (-50.5, 0.5) to another Gaus- sian (0.5, 0.5).
Left:
Step 3.(Resmmllngg) Draw x" from the discrete distribution over { Tn. with sampling prob- abilities p(ara) w(x:)/ Σド= w(xy).
SIR is very similar to importance sampling(IS), except that IS samples according to g and multiplies the result by the importance ratio, while SIR corrects the distribution by the importance ratio to approximate p. The following proposition shows the consistency of SIR: + oc, the resampling distribution converges to the target distribution
Proposition 1. lfsupp(g) 2 supp(p), as - oc, the samples from SIR will consist of independent draws from p. Namely, As - x² is distributed according to p.
All proofs could be found in Appendix B
Fig. T illustrates SIR for distribution correcting simple one-dimensional cases. Note that Fig. (left) simulates the case when a dataset is generated by uniform distribution and the target distribu- tion Gaussian, while Fig. (right) corresponds when a dataset generated by some Gaussian distribution, and the target distribution is Gaussian with different mean.
4.2 IN-SAMPLE ACTOR CRITIC
In this work, we adopt SARSA-style in-sample learning because in-distribution constraints widely used in prior work might not be sufficient to avoid extrapolation error (Kostrikov et al. 2022)
Our method is based on policy iteration, which consists policy evaluation (PE) and policy im- provement (PI). In PE, using in-sample actions a' 'D rather than a' ~ ((-Is') in the TD target introduces bias. We consider introducing an importance ratio wis's a') = π [Ta'ls (') (3aa|s') Un- der the assumption of importance sampling: A(r.)T(a(1s)) can be rewritten as follows
whenever ((a]]s()
Eqn.
ATD (. Bad (o oooVVool
Q ())))
(4)
Here, the assumption of IS (as well SIR) actually coincides with the conception of support- constrained policy set (Kumar et al. 2019):
Definition (Support-constrranee policy). Assuming the data distribution is generated by a behav-
ior policy B, the support-consttainned policy class IT is defined as
{TT (als)) = =0 whenever B(als) 0}
(5)
This means that a learned policy [a]ss) has a positive density only where the density the behavior policy B(a]s) is positive, instead of the constraint the value of the density (a]ss) and B(ass)
which is overly restrictive in many cases. Previous works have demonstrated the superiority of restricting the support of the learned policy (Kumar et al. 2019 Ghasemipour et al. (021)
In practice, is unrealistic to use the whole dataset to empirically estimate ATD (expected value of update) every iteration, in spite of its low variance. Consequently, consider estimating ArD by
4

---

Published as a conference paper at ICLR 2023
sampling mini batic of size lo from the dataset. Specifically, sample fø1...., R,), where ž, is sampled uniformly from D (I1,...,,2y It leads to an IS estimator of ATD:
iIS = c((s)))))(())) By fa Fnf uniformly
(6)
Though IS is consistent and unbiased (Kahn & Marshall 1953).
it suffers from high even infinite
variance due to large magnitude IS ratios (Precup et al. 2001 The high variance of Åis could destabilize the TD update and lead to a poor solution.
In this work, we adopt SIR instead of IS to reduce the variance and stabilize training, Specifically, we remove the IS ratio and sample { where é, is sampled from f.i...., Tn with probability proportional w w(ss, ())) rather than uniformly like all prior offline RL works. It leads to another SIR estimator of ATD:
& ÂSIR Σ (8t), 1=1
w0 al))
By fx1,....... [[fi,...,,,,, 2n) with probability Pj = Ei=1 (())
(7)
Intuitively, in offline RL setting, this resampling strategy reshapes the data distribution of D to adapt to the current policy. Unfortunately, unlike the IS estimator Åis, ÂSTR is biased estimator of ATD- Subsequently, we show that by simply multiplying ÁSTR with the average importance ratio the buffer w == A Ei Wi, we get an unbiased estimate of ÂTD.
Theorem 3. Assume that an offline dataset 'D of n. transitions is sampled ii.d according to ps(z= as, a')) = PP a)B((a(s'), and π is support constraico (i.e. π E II³). Then,
E ElDASIRl ATD
where ATD is the expected update across all transitions in 'D defined in Eqn. empirical update across the sampled mini-batch defined in Eqn. (7 w Diz W. is the average imporiance ratio in the dataset.
2 ASIR is the
In fact, ÂSTR gives the correct direction, and we do not need to care about the actual value of the update. The reason is that, the scalar w remains the same across all mini-batches during SGD learning, so can include to in the learning rate and just use Âsın as the update estimator in PE. We point out that there is no need to adjust the conventional learning rate, hecause for large enough dataset, is close to
Theorem 3 guarantees that if the policy π is constrained within the support of behavior policy B during learning, our method yields an unbiased policy evaluation process via in-sample learning, thus avoiding extrapolation error. Conversely, the support of the current policy deviates much from the dataset, which common practice when the dataset distribution is narrow, and the trained policy is randomly initialized. Theorem 3 can not provide performance guarantees. So in PI, we implicitly enforce constraint with advantage-weighne Peng et al.] 2019), controlling deviation from the behavior policy. Since in PE, we have sampled transitions _ } non-uniformly from D, for convenience, we use the same transitions to perform PI, instead of sampling from D again uniformly, leading to the following loss:
regression (Peters Schaal, 2007
(Ø) (e,) [exp(8(Qe(5) a) - Ea-ra (s) Qe(s, a)) log TT(a[s)].
(8)
where & denotes sampling from discrete distribution p.
Note that even though our method is in-sample, constraint (implicit or explicit) is necessary due both our theoretical requirement and empirical results of previous works. Among previous in- sample approaches, IQL (Kostrikov et al., 2022) adopts the same advantage-weighted regression in the policy extraction step, while the performance of OneStep RL (Brandfonbrener et al., 2021) will have sharp drop without constraints. One possible reason is that in in-sample methods do not update out-of-sample (s, a) pairs. Their Q-values are completely determined by the initialization and generalization of the neural network, which is uncontrolled. As result, despite that in sample methods address the Q-value extrapolation error, vanilla PI without constraints can still choose out- of of-sample actions whose Q-values very inaccurate.
20 Bandd (())))
T(als)
Σ.
r(als) Pajs 3(a ((a)s)da(s)
D(
5

---

Published as a conference paper at ICLR 2023
4.3 LOWER VARIANCE
Theorem 3 shows that the proposed method provides unbiased policy evaluation with in-sample learning. In this section, we theoretically prove that our SIR estimator (see Eqn. 7 has lower variance than the IS estimator in many cases and thus yields a more stable learning process.
Proposition 4. Assume that the gradient is normalized, then the following result holds,
WiwAsial S V[Ars).
Proposition 4 indicates that when the scale of the gradient for the sample does not vary lot across the dataset, there is a high probability that SIR will have a smaller variance than IS.
Proposition 5. Assume that [1)()] has positive correlation with B(a'ls") for E 'D and policy is uniform, then the following holds.
VIAASIR S v[Års).
(10)
In general, the sample with large behavior distribution density usually has small-scale gradient due to training, which corresponds to the assumption in Proposition 5
4.4 PRACTICAL ALGORITHM
IAC is the first practical algorithm to adapt the dataset distribution to the learned policy, and design the algorithm be simple pos- sible to avoid some complex modules confus- ing our algorithm's 's impact on the final perfor- mance.
Algorithm IAC
Input: Dataset D = {(s, a, a.r, r',rs'))) Initialize behavior policy Bw,, policy network
Té. Q-network Qu. and target Q-network Qp
Density Estimator. IAC requires the behav- ior density 3 to be the denominator of impor- tance resampling weight (Eqn. (7)). We learn a parametric estimator for the behavior policy by maximum likelihood estimation. The esti- mated behavior policy ih is parameterized by Gaussian distribution. The objective of Ba is
Behavior Policy Pre-training for each gradient step do
Sample minibatch (s, a) DD
Update according to Eqn. 11 end for
// Policy Training
for each gradient step do
Sample minibatch (s, T, 8',a') propor-
max MaxxEaarp log (als),
(11)
tional to p(s', a') from D
Pw
Update 0 by applying TD update in Equ.
where w is the parameter of the estimated be-
Update & by minimizing L (0) in Eqn. 8
havior policy.
Update target network: O' <- (1 0 7
Policy Evaluation. In policy evaluation phase, IAC uses non-uniformly sampled SARSA (sampling proportional to p(s', a') to evaluate the trained policy. We represent the policy with Gaussian distribution for its simple form of density.
end for
Policy Improvement. In policy improvement phase, Eqn. 8 requires calculating the expectation of the Q-value concerning the current policy. We find that replacing the expectation with the Q-value of the policy's 's mean already obtain good performance. Also, it simplifies the training process without learning V-function.
SumTree The importance resampling weight is determined by <(a'ls') and B.g (a')s'). While B (a')s') is fixed after pretraining, TTa'ls') changes as updates during training. We adopt the Sumtree data structure to efficiently update during training and sample proportional to p. It is similar to prioritized experience replay (PER) (Schaul al.) 2015). In PER, p is implemented as the transition-wise Bellman eтror, where sampling proportional to P replays important transitions more frequently, and thus the Q-value is learned more efficiently. In our proposed algorithm, is implemented as the importance resampling weight. Sampling proportional p provides unbiased and in-sample way to evaluate any support-constrained policy.
Overall algorithm. Putting everything together, summarize final algorithm in Algorithm I Our algorithm first trains the estimated behavior policy using Eqn. (IT) to obtain the behavior den- sity. Then it turs to the Actor-Critic framework for policy training.

---

Published as a conference paper at ICLR 2023
5 DISCUSSION
5.1 ONE-STEP AND MULTI-STEP DYNAMIC PROGRAMMING
The most significant advantage of one-step approaches (Brandtonbrener et al. 2021 is that value estimation is completely in-sample and thus more accurate than multi-step dynamic programming approaches, which propagate and magnify estimation errors. On the other hand, multi-step dynamic programming approaches can also propagate useful signals, which is essential for challenging tasks or rewpperforaace datasets. IAC belongs to multi-step dynamic programming and enjoys the ben- efit of one-step step approaches.
To show the relationship between IAC and one-step approaches, we define general SIR simulator:
w(s',. af)"
À SIn
(dj)) ==1
3j (xi, ( Inl with probability Pj Σ u066 af)"
(12)
Note that IAC corresponds to the case when 17 while it reduces to OneStep RL (Brandfonbrener al. 2021 when 1j = We have the following result about Âii
Proposition 6. Assume that Vo E D, ((x) = h, where h is constant vector. Let n E [0, (), Wn denote 1 - a. Assume that EØ=1 1088 ag) 2 n, then the following holds
Vi AS S V[wAsin).
(13)
It indicates that 7) < 1 might bring smaller variance when Axx) is the same for all E D. However, might not be the case and introduces a bias when (x) varies across the dataset. In our experiment, we show that the performance of choosing Å is better than that of choosing Åg which indicates that reducing the bias matters for resampling.
5.2 OTHER CHOICES IS
Other than Eqn. 4 reweighting the gradient of value function is an alternative choice to utilizing IS. The TD update of value function could be written as follows:
ATD B(s.a,')~p(u((s a) V Va (V)(() r(s,a) 7Va('')))
(14)
We point out that a Q-value function is still required and learned via Bellman update to learn a policy. This implementation increases computational complexity compared to IAC. In addition, learning three components simultaneously complicates the training process.
EXPERIMENTS
In this section, conduct several experiments justify the validity of our proposed method. We aim to answer four questions: (1) Does SIR have smaller variance than IS? (2) Does our method actually have a small extrapolation error? (3) Does our method perform better than previous methods on standard offline MuJoCo benchmarks? (4) How does each component of IAC contribute to our proposed method?
Varance office
6.1 VARIANCE
We first test the variance of SIR and IS on two-arm bandit task. The first and the second action's 's reward distributions are 1),111 and (11,, 1), respectively. We fix the dataset's s size to 100, 000. Then we vary the ratios of these two actions' samples in the dataset to simulate a set of behavior policies. For policy that chooses the two arms with identical probability,
Figure 2: SIR has smaller variance than IS on a two- arm bandit task.
evaluate the policy with SIR and IS based on each behavior policy in the set. Fig 2 shows the variance of SIR and IS when provided with different behavior policies. According SIR has smaller variance than IS, especially when the dataset is highly imbalanced.
to Fig 2
6.2 EXTRAPOLATION ERROR

---

Published as a conference paper at ICLR 2023
Table 1: Averaged normalized scores on MuJoCo locomotion on five seeds. Note that m-medium, m- r-medium-replay, r-random, m-e=medium-exper,, and e=exper.
BC
OneStep
ID3+BC
COL
IAC-w/O-B
halfchoetah-m-v2
42.0=177
50.4
48.34:0.3
47.0=0.5
4.22=0.
52.210.3
52.0:0.6
51.40.3
hopper-m-v2
56.2443
87,5=10.9
59.3::4.2
53.0+28.5
66.2-5.7
83.1=234
6384998
14.6=11.5
walker2d-m-v22
71.0:6.5
848429
83.742.1
73.3
78.348.7
83.6
85.340077
85.24:.4
halfehootal--n-444222
36.4227
42.7
44.6::0.5
43.5=007
4..2
47.5++0.5
47.9400
47.2-20.3
hupper-m-t-v2
98.54:27
60.911888
88.74-2299
94.7188.
102.7:1.22
99.3:227
103.2:1:00
walker2d in-r m22
24.916.3
61.71:16.3
81.8.5.55
81.8.2.77
73.8.17.11
93.2.100.
91.1.1111
93.2.1.88
halicheetah-m-co
59.64...8
90.744.33
75.64225.7
86.7..5.3
89.6:12.55
hopper-m-o-v2
51.7=2.44
108.6444.1
98.0+99.4
105.6=1229
11.3+114.3
111.0=1116
107.8:599
109,3+4,00
walker2d-m-e--2
101.2+3366
111.3494
110.1::0055
107,9=166
109.6=1.
113.3::12
109,::0.88
110.1=0011
halfehectah-o-v22
92.9=0.5
88.24655
%6.7111
96.3=1.3
95.0=0.5
94.7:100.4
94.4:10.3
94,5
hopper-e-V2
110.9:0033
106.9:4411
107.87
96.5+-2800
109
110.7:::17
111.6::0.2
110.6=19
walker2d-ene
109.6:10.11
halfcheetab r.v2
2.6:1.00
2.3-10.0
11.01111
1755
1.31
23.2:122.3
21.91110
20.9
hopper-r-v2
4.1+0.11
5641166
8.5-20066
79-004
79-00.
31.5+0033
27.6-9.11
31340.33
2alker2d---v2
1.2+0.0
6.9122
1.+177
5,1=13
5,4=122
4.0-1999
2,4+0.3
3.0
locometionn.e
In this part, we compare IAC baseline that replaces the resampling section with the gen- eral off-policy evaluation, which is updated by
lepeosteiion
10.
IAC 14C-SE
10
IAC-Q vdae
Eqn. ( The policy update and the hyper- parameters for IAC and the baseline are the same. Although
10".
Hlndine:
19'
-rrranttge-wigghee regres- sion enforces constraint on the policy, the baseline might encounter target actions that are out of the dataset. We experiment with both methods four tasks in D4RL (Fu al. 2020). We plot the learned Q-values of IAC and the baseline in 3 Also, show the true Q-value of TAC by rollouting the trained policy for 1, 000 episodes and evaluating the Monte-Carlo return. The result shows that the learned Q-value of IAC close the true Q- value. Note that the learned Q-value is smaller than the true Q-value on walker2d-medium and halfcheetah-randonn tasks. The reason is that taking the minimum of two target networks will lead underestimation. By contrast, the Q-value of the baseline increases fast and far larger than that of IAC. It indicates that proposed method has lower extrapolation error by only referring the target actions the dataset.
19t
(X205)
bultterah
107
10
14
10'
Gradier Stque (x10*)
Gradier Stope (x20*)
Figure 3: of IAC and baseline without resampling.
True Q-value of IAC, learned Q-values
6.3 COMPARISONS ON OFFLINE RL BENCHMARKS
Gym locomotion tasks. We evaluate our proposed approach on the D4RL benchmark (Fu et al., 2020 in comparison prior methods (Table We focus on Gym-MuJoCo locomotion domains involving three agents: halfcheetah, hopper, and walker2d. For each agent, five datasets are provided which correspond to behavior policies with different qualities: random, medium, medium-replay. medium-expert, and expert.
AntMaze tasks. We also compare our proposed method with prior methods in challenging AntMaze domains, which consist of oparse-reward tasks and require "stitching" fragments of suboptimal tra- jectories traveling undirectedly to find a path from the start to the goal the maze. The results are shown in T
Baselines. Our offline RL baselines include both multi-step dynamic programming and one-step approaches. For the former, compare to CQL (Kumar al. 2020). TD3+BC (Fujimoto & Gu, 2021). and IQL (Kostrikov et al. 2022). For the latter, we compare to OneStep RL (Brandionbrener
al.
2021
8

---

Published as a conference paper at ICLR 2023
Table Averaged normalized scores on AntMaze on five seeds. Note that u-Umaze, u-d=Umaze- diverse, m- m-p-mediuu-rrplayy m-d-medium-diversy l-p-large-replay and A-dllargeddivers,
OneStep
TD3+BC
COL
IQL
IAC-wo-BB
IAC4S
ontmaaz-u--2
66.846.7
54,023,4
73.0E34.0
82.6=3,7
89.614.22
71.6
0.0:0.0
77.64:3.8
antmaze-u-d-22
56.842266
57.8+14.0
47.027.3
10.2=6.7
65.6:28.3
52.4=7.2
0.000.0
71.24:..66
unmaze-m----2
0.0:0.00
00:0..
0.0+0.0
59.0=1.6
76.4:::777
75.0=2.7
33.0:15.6
72.0:4:66
animazz-m-d-12
0.0:0..0
0.6:0..5
0.2:0.4
46.6+:24.0
72.8+7.00
67.2=6.9
17.2422..
74.24-4.1
unimaze-l-p-12
0:0:::000
0.0:0.00
0.0::000
16.4:117.1
42.622388
34.0:1288
57.0-1.444
animaze 1d.22
0.0:1.0.0
0.2.1.04
0.0.1..00
3,2.14,1
1..5
38.8.1
0.0.1..00
47.2.19.44
unimaze-vV total
123.6
112.6
120.2
218.0
392,4
347.6
84.2
399.2
Comparison with baselines. On the Gym locomotion tasks, we find that LAC outperforms prior methods. On the more challenging AntMaze task, IAC performs comparably to IQL and outper- forms OneStep RL by a large margin.
Comparison with one-step method. Note that one-step method corresponds to Å which samples uniformly. In the AntMaze tasks, es- pecially the medium and large ones, few near-
bulkterak
optimal trajectories are contained, and the re-
spodde Raam
ward signal is sparse. These domains re-
quire "stitching" parts of suboptimal trajecto- ries find path from the start to the goal of the maze (Fu et al. (020). Therefore, one- step approaches yield bad performance these
0.3
(x205)
challenging domains where multi-step dynamic
agassa Roou
programming is essential.
We point out that using ÂLTR gives IAC the power of multi-step dynamic programming. At the same time, inheriting the advantages one- step approaches, IAC uses in ia-sample data, thus having a low extrapolation error. As shown in Fig/4] our proposed method performs much bet- ter than choosing = 0, which corresponds to OneStep RL.
%00
Gradion (xtu
Godden Sapee1101
Figure 4: Comparison with OneStep RL (17 0) and importance sampling.
Comparison with importance sampling. We refer to the algorithm which updates the Q-value importance sampling(seen in Eqn. (4) IAC-IS, and we test its performance on MuJoCo and AntMaze tasks. The result is shown in Table Table and Fig IAC-IS performs worse than IAC slightly on Gym locomotion tasks. For the challenging AntMaze tasks, there is large gap between the two algorithms. IAC-IS even obtains zero rewards on half of the tasks. The reason might be that IAC-IS has larger variance than IAC, which would impair the learning process.
Ablation on the estimated behavior policy. IAC requires access to a pre-trained behavior policy, which brings a computational burden. Removing the behavior policy and regarding the behavior policy density as constant will introduce a bias but reduce the computational load. We refer to this variant as IAC-w/o-3. As shown in Table [ and Table IAC-w/o-B could still obtain desirable performance most Gym locomotion tasks and several AntMaze tasks. Thus, IAC-wo-$3 is an appropriate choice for its lightweight property when the computational complexity is a priority.
7 CONCLUSION
In this paper, we propose IAC to conduct in-sample learning by sampling-imporrance resampling. IAC enjoys the benefits of both multi-step dynamic programming and in-sample learning, which only relies on the target Q-values of the actions in the dataset. IAC is unbiased and has a smaller variance than importance sampling in many cases. In addition, IAC is the first method adapt the dataset's distribution to match the trained policy dynamically during learning. The experimental results show the effectiveness our proposed method. In future work, we expect find better estimated behavior policy to boost our method, such as transformers (Vaswani et al. 2017).

---

Published as a conference paper at ICLR 2023
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- bassed offline re- inforcement learning with diversified g-ensemble. Advances in neural information processing systems, 14:7436-744,, 2021.
David Brandfoobrener, William F Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-po evaluation. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: /ttEp::://ppnneviee.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl -l with im- portance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing funetion approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected- q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp. 3682-3691. PMLR, 2021.
Neil J Gordon, David J Salmond, and Adrian FM Smith. Novel approach to dealinearr/nongaussiii bayesian state estimation. IEE proceedings F (radar and signal processing), volume 140, pp. 107-113. IET, 1993.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Process- ing Systems, 33:7248-7259, 2020.
Herman Kahn and Andy W Marshall. Methods of reducing sample size in monte carlo computations.
Journal of the Operations Research Society of America, 1(5):263 278, 1953.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.666 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. 13/266 14, 2013.
arXiv preprint
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations, 2022. URL https://
openreview feaett/formmmiid 63n2s229uu3
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q- learning via bootstrapping error reduction. Advances Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 1991 2020.
10

---

Published as a conference paper at ICLR 2023
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pp. 6120-6130. PMLR, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprini arXiv:2005.0164: 2020.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off-policy estimation. Advances in Neural Information Processing Systems, 31, 2018.
A Rupam Mahmood, Hado P Van Hasselt, and Richard S Sutton. Weighted importance sampling for off-policy learning with linear function approximation. Advances in Neural Information Pro- cessing Systems, 27, 2014.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 8(77540):529-533 2015.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off- policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of dis- counted stationary distribution corrections. Advances Neural Information Processing Systems, 32, 2019.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. wevintage-weiigtted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 1rXiv:1910.00117,, 2019
Jan Peters and Stefan Schaal. Reinforcement learning by seward-weightn regression for operational space control. In Proceedings of the 24th international conference Machine learning, pp. 745- 750, 2007.
Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-policy temporal-difffrence learning with function approximation. In ICML, pp. 417-424, 2001.
Martin Riedmiller. Neural fitted q iteration-first experiences with data efficient neural reinforce- ment learning method. In European conference on machine learning, pp. 317-328, Springer, 2005.
Donald B Rubin. Using the sir algorithm to simulate posterior distributions. Bayesian statistics, 3: 395-402, 1988.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, arXiv preprint arXiv:1511 1it.:09911..0.. 2015.
Matthew Schlegel, Wesley Chung, Daniel Graves, Jian Qian, and Martha White. Importance resam- pling for off-policy prediction. Advances Neural Information Processing Systems, 32, 2019.
Øivind Skare, Erik Bølviken, and Lars Holden. Improved sampling-importaacee resampling and reduced bias importance sampling. Scandinavian Journal of Statistics, 30(4):719-737, 2003.
Adrian FM Smith and Alan E Gelfand. Bayesian statistics without tears: a sampling-resappling perspective. The American Statistician, 46(2):84 88, 1992.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in newral information processing systems, 28, 2015.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.

---

Published as a conference paper at ICLR 2023
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang. and Mingsheng Long. Supported policy opti- mization for offline reinforcement learning. arXiv preprint arXiv:2202.062.3, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1911:1111..1361, 2019.
12

---

Published as a conference paper at ICLR 2023
A SAMPLING METHODS
The problem is to find [xf(x) = Jp (x))((a)dz where is the target distribution on D C R when only allowed sampling from some proposal distribution g on D. Define importance ratio w(x) p(x)//(x;)
A.1
IMPORTANCE SAMPLING
condition: ((x) whenever f(x)p(x) + O, ie.. supp(4) supp(p f); E.g (xx)f(x) < too
[ Jaapetal
(()) [(x)
Ep(f(x))
f(x)dr = B, [f(x)w(x))
IS estimator of J::
1 PIs
(x,))((xss ~0-9
bias:
Big (fis) = 1
variance:
Var, Varg(f (f()((())
Var, (Ais)
n / (f(x)p(x))
g(x) dr 42
1 (f(x))(x)) p²(x))2 n
dx
((x)
How to select a good proposal p? The numerator is small when f(x)p(x) - pq(x) is close to zero, that is, when q(x) is nearly proportional f(x) ((z)) From the denominator, see that regions with small values of ((x) greatly magnify whatever lack of proportionality appears in the numerator.
Theorem (Optimality Theorem). For fixed n, The distribution q that minimizes the variances of jis
|s(x)p((z)
[lf(t)p((addr ({{())p((z)
A.2 SAMPLING-IPPORTANCE RESAMPLING
Step ((Sampling) Draw an independent random sample (z1...., x,, from the proposal distribution
Step 2. (1mmorttace)) Calculate the importance ratio for each oy: w(xx)) = p((x;)/((x))
Step ((eesampllng) Draw x² from the discrete distribution over (æ1,..., Fn with sample proba- bilities, p(x;) wi/ Σラ Wy.
A.3 BATCH SETTING
Question: If the problem is to compute Exf(x), which is better, IS, SNIS, SIR? Assume IS and SNIS have sample {=1,-... Wn while SIR resamples k items batch i = (ā1,- ........ Fx. from B with probability A; proportional to w;. For fair comparison, we also consider batch version IS-b and SNIS-b, which resample à items i = { &x} from B uniformly. The estimators are follows,
A1s = Σ ((x))((1)))
13

---

Published as a conference paper at ICLR 2023
&
AIS-b =
((;)) (T:)
i=1
2im1 f(x)) Σ w(r:)
ASNIS
Σ ((t;)
BSNIS-b-
Ei= m((is)
ASIR Σ f(5,)
Proposition 8. PSIR has the same bias as PSNIIS. Proof.
Bolosil E 1
Σ
f(a:))
j=1
W
EB
f(x;)
2===1 Wi
EBSgRSIIS
Note that if and are normalized, ÜĀSIR is unbiased, where i = - w (T:) Proposition 9. ĀSIR is consistent as
For variance, compare the unbiased WĀSIR and Åis-b. On the one side, fis and ÅSNIS use entire dataset B that have much more items than batch and should have low-variance estimate. For another, PSNIISb baised and p and q are normalized, the bias-corrected version of PSNIS-b is just Prss-b-
Proposition 10. For fixed B. let μB Ep]f(x)(B). The variance of wÍSII and Pis-b follows.
Var(Ors-o|B)
Σ
w(,,)
MBPB
n
1 k n
1
Σ
my 11 - MTAB
Proof. Since we condition on the dataset B, the only source of randomness the sampling mecha- nism. Each index is sampled independently have that,
Var(wüsrR B)
Var (w](Tx) Var (w B)
12
and similarly
Var (ar(Ais-o\B) = (at(
14

---

Published as a conference paper at ICLR 2023
We can further simplify these expressions. For the IS-b estimator
Var(Ars-o/ Var(w((1)
Var(u({:)f(;1)1BB)
k
(Blw((1))]((() ((=1) B] B(w((11)((11) BJ (lv(()1)((11)) B))
(E(w(()) (1.(())) ) B (gPB)
since (&1).f())) B is unbiased for pB
n ((fx;)) PTuB
For the SIR estimator, recalling that v = 1 E W/ we follow similar steps,
Var (if()
(E[w² f(i1) f B( -(()((1) Bl"
|B])
Blw [[ ) B pbUB)
since ff B unbiased for μc
"
11((z;)11 u
E wi
wy [1,(2;;)1 MBuB
Under what condition SIR estimator has lower variance than IS-b?
Condition 1: [xx > c/w; for samples where wy 2 w. and [[;(2) < c/w, for samples where wy < w. for some e 0.
B PROOFS
B. PROOF OF PROPOSITION I
Sampling-importance resampling (SIR) aims at drawing random sample from target distribution Typically, SIR consists three steps:
Step (SSmpllngg) Draw independent random sample (.1...., Fn) from the proposal distribution
Step 2. (Importance) Calculate the importance ratio for each zę: w(xx) p(x:://((z))
Step 3(Resampling) Draw x² from the discrete distribution over ( 2r with sample proba- bilities, p(x;)) wt/ Σ}= Wj-
15

---

Published as a conference paper at ICLR 2023
Proof. x² has edf
Pr Pr(x" S x0)
pillas (-00,
==1
1 Mi= Wi [2; E (-)0,
Bg P E (-oc, Ba
-i=1 w/
20
p(z)dz
00
p(
Note that even p and are unnormalized (but can be normalized), this method still works. The sample size under SIR can be as large desired. The less p resembles the larger the sample size n will need to be in order that the distribution of x" well approximates p.
B.2 PROOF OF THEOREM
3
Proof. Note that there are two source of randomness for the estimator ÂSTR. First D = (f1.-.., } is sampled i.i.d. according Pg- Second, our method draws i.i.d. from the dis- crete distribution over kn) placing mass pt on Is, forming a mini-batch , }
k
ED Br (uspll = WED-ng E
A A(t;) = W.D ((T))
W A(r;))
WEDapp
P:((T;) = WED-pp
プド= Wy
1=1
1=1
EDapp
wc wx)(x;) = -Ezup, (axx))
('als')
Pr
Euwps
A(x) = Eanps (z(x)
A(x) = =Egxp, A(z)
3(a'(s')
ATD
B.3 PROOF PROPOSITION
- "āua). where i is the size of the batch and μB the expectation of the estimator. Since SIR is unbiased, we have V[wâsın EE Wy || (x;))) PBUBB) for SIR.
Proof. For IS, we have v = & ( "
Now the problem is to show that (
Σラ= ((3;))
((,))9A((>)11122 w Σ ((,)))
(15)
Σ P ((
(16)
5=1
where P5
w(x) E.- w(.)
w(x)
16

---

Published as a conference paper at ICLR 2023
Assume that normalized gradient is applied: [1) = 1. Then according to Inequality, Eqn. (16) hold.
n
1
x Pj
= =1
Pj
1
B.4 PROOF OF PROPOSITION
Assume that (Z),... .n.) is in descending order in terms of || (z;;) 16 otherwise could rearrange these items. Considering that [>) is in positive relation to 3[ā|)] for x E and
Proof.
policy is uniform. To simplify notation, we use [z)() to denote || (x) We have
[| 2 [ 2 ... 2 [1
(17)
M 22 P2 2 Pn. (or w(x) w(x2)) 2 ... 2 w(xn))
(18)
Let denote by j. the index of largest Pj satisfying P5
1 i.e.
j. min { j j
1 : و S n
n
is easy to show j. > 1. We have Pj - 2 0 when S j S j.
and P3 - < when
j. S j S Then
Σ pallf(a))1?
p0 - n
1
Σ
p=llf(=;)11
(3 (s--
1 ))
Σ p,llf(=;))1 p3 ())-;))
j=
2
Σ
Ps_ Pa--1 P)-1111(3;,->)1 (0)
1
+ Σ
pj
05 (
j=1
j=
Pp.- -11-
Σ
Pj
j.
+
Σ
(03 (
n
5=
j=
Σ Pj. [|1 (1)
(= - P3 p3)
+ Σ
p)|l((a))11
(0g -
Σ (1
1
Ps
20.
B.5 PROOF OF PROPOSITION 6
Assume that Vo E D. ((x) = h, where h is a constant vector. Let n E [0, 1]. W. denote 1 E w83 a)),, Assume that En w(3j, az) 2 n, then the following holds
Vi A L < V[WÂSıR].
(19)
Proof. we have V[wASın) = ( Σー myy (((3)))))
Vi Ag = En Σy w/ (())))))))
17

---

Published as a conference paper at ICLR 2023
When normalized gradient is applied, the problem is to show that
n1
W
W
71
Wj
w
==1
j=1
According to Holder Inequality, we have
1-1
7
15
Tj
1=1
j=1
a
)
n
1
3j
j=l
By choosing Iÿ w/ and ;/, we have
n w
1-
According to the condition ΣØ=1 Wj 2 The following holds,
W 1
W n
C EXPERIMENTAL DETAILS AND EXTENDED RESULTS
C.I EXPERIMENTAL DETAILS
Table 3: Hyperparameters of policy training IAC
Hyperparameter
Value
Optimizer
Adam (Kingma & Ba 2014
Critic learning rate
3 x 10
Actor learning rate
3 x 10
with cosine schedule
Batch size
256
Discount factor
0.99
IAC
Number of iterations
10
Target update rate т Policy update frequency
0.005
2
Inverse temperature of AWR
{0.25, 5) for Gym-MuloCo
(10) for AntMaze
Variance Gaussian Policy
0.1
Actor Critic
25pt-256-226-ourr
Architecture
input-256-256-1
For the MuJoCo locomotion tasks, average returns of over 10 evaluation trajectories and ran- dom seeds, while for the Ant Maze tasks, we average over 100 evaluation trajectories and 5 random seeds. Following the suggestions of the authors of the dataset, subtract from the rewards for
18

---

Published as a conference paper at ICLR 2023
the Ant Maze datasets. We choose TD3 (Fujimoto al., 2018) mize a deterministic policy. To compute the SIR/IS ratio, we need the density any action under the deterministic policy. For this, assume all policies are Gaussian with fixed variance 0.1. Note that IAC has no additional hyperparameter to tune. The only hyperparameter we tuned is the inverse temperature 3 in AWR for PI. We use 3 = 10 for Ant Maze tasks and {0.25, 5) for MuJoCo locomotion tasks (3 = 0.25 for expert and medium-exper datasets, & = 5 for medium, medium-replay, random datasets). And following previous work (Brandfonbrener al. (2021). clip exponentiated advantages to (oo0, 100). All hyperparameters are included in Table
base algorithm and opti-
C.2 LEARNING CURVES OF IAC
widdeaeeddoooooed 100 Mecheeeeddar
0131 315 110 70
VTT
EST 1115
12.5 055
435 131
100 135 013
(x20
Godier Stpt
Grader: Sape 10")
Crodatt (x10%)
Oradett (x10))
hopperddeed
Mun 5
5
Guddan (xp0(
Godica (K10%)
Gradie Supe (0207)
Coudaar RoaM (X107)
Gradiex Exppt
wedket enperrad
vellect
103
Telaco harre
12
1000 025 a:d a7t 100 Magee 2011
9as 6az Ede a.5
2.0 0.5 058 355 1.00
a35 6:50 6:7 Crodiie
Gradee Stpp
UradenSiggg (N
Figure 5: Learning Curves of IAC on MuJoCo Locomotion Tasks.
D ABLATION ON BEHAVIOR POLICY
D. VAE BEHAVIOR POLICY
In this section, we conduct ablation study on behavior policy. Like previous works (Fujimoto al., 2019 Wu et al. 2022). we consider to learn the behavior density 3 explicitly using conditional variational auto auto-encoder (Kngmma & Welling 2013 Sohn et al. 2015). Specifically, {(a(s) can be approximated by Deep Latent Variable Model Pw, (a|s)
Pw,, (a|z, (()2s)d2 with prior p(z[s) = (0, 7). Rather than computing Pw, (a|s) directly by marginalization, VAE construct a lower bound on the likelihood Pa,, (als) by introducing approximate posterior gos (2la, s):
Pay (a, 2|8)
Paj 2|s)
log Pw,y (a|s) log Reg,, (x)a,s)
Eyng (sla.s) log qwo (2la,
awn (2/a, s)
Eyn s))a,)) a) [log Pwi (a/2, 8)] KI. (gwn (z/a, s) [pp{(ss)
(20)
def JE,BO aw
converts the difficult computation problem into an optimization problem. Instead of maximizing the log- logeliieldoood log Pw; (a]s) directly, now optimize parameters W1 and wa jointly by maximiz- ing the evidence lower bound (ELBO) (ELBO((S a;w). After pre-training the VAE, we simply use JelBo(s, a;w) approximate log (a]s) in Eqn. (7)
19

---

Published as a conference paper at ICLR 2023
100
animuze umnaze-v2 +2
100
un unimaaze umazz diverse-v2
100
unimaaze medium medium-play-v2
Epicade Return
Episode Return
Episode Return
50
50
50
0.00
0.25
0.50
0.75
1.00
0.60
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
Tib
Gradhent Steps
(x109)
Grudient Stepss
(x109)
Gradient Steps (x109)
unimusee celmmae--mdiim--dee pecdiiun-diverre---
unimaze largellagpeea -2
antmace untmaze-large diverse-v2
100
Episode Retur
Episode Rotum
Lipisude Retum
50
50-
0.25
0.30
0.75
1.00
0.00
0.25
0.50
0.75
0.00
0.50
0.75 1.00
Gradieet Steps (x10%)
Gradiont Steps (x10%)
Gradient Sceps (x10%)
Figure 6: Learning Curves of IAC on AntMaze Tasks.
Table 4: Averaged normalized scores on MuJoCo locomotion on five seeds. We compare IAC with IAC-MM(LAC with multi modal behavior policy), IAC-SR(IAC with state ratio), IAC-VAB(IAG with VAE behavior policy), and IAC-SNIS(IAC with self normalized im- portance sampling). Note that m=medium, m-r=medium-replay, r=randmm, m-e=medium- expert, and e=expert.
Dataset
IAC-MM
IAC-SR
IAC-VAE
IAC-SNIS
IAC
halfcheetah-m-
51.9+0.0
51.410.3
52.2+0.2
51.3+0..
51.610..
hopper-m-v2
88.9+12.7
61.5+0.3
91.9+18.3
82.9+10.2
74.6+11.5
walker2d-m-v2
84.8+0.7
84.7#1.0
85.2+0.5
82.8+0.7
85.2+0.4
halfcheetah-mm-rv2
47.5+0.00
46.4+..00
47.6100.4
47.940.6
47.2+00.3
hopper-m-r-v2
103.9+0.2
100.9+1.1
103.1+1.11
99.9+1.0
103.2+1.0
walker2d-m-r-v2
90.9+1.0
92.7#116
93.610.4
92.7+0.3
93.2+1.8
halfcheettah-m-e-v2
92.7+0.4
94.7#1..
87.817.4
87.149.2
92.910.7
hopper-m-e-v2
108.3+0.0
109.4+0.5
111.6+0.5
107.8+00.7
109.3+4.0
walker2d-m-e-v2
110.2+0.1
110.140.1
109.9+0.5
109.7+0.4
110.1+0.1
halfcheetah-e-v2
94.6+0.0
94.5+0000
94.6+00.3
94.9+0022
94.5
hopper-e-v2
111.0:0..5
111.110.1
111.40.44
111.140.0
110.6
walker2d-e-v2
109.8
109.440.1
109.640.1
109.740.2
114.8
balfcheetah-r-v2
21.3+0.6
19.240.8
21.010.77
23:0.99
20.9
hopper-r-v2
20.4+11.1
19.219.5
31.3+0.0
3240.2
31.310.3
walker2d-r-v2
0.4+0.2
7.5+7.7
1.3+2.3
0+0.00
3.0+1.3
locomotion-v2 total
1135.8
1112.7
1152.1
1132.8
1142.4
We term this variant VAE. The results of IAC-VAL are shown in Table Table Benefiting from the VAE estimator, IAC-VAE obtains better results in MuloCo locomotion tasks.
D.2 CATEGORICAL BEHAVIOR POLICY
Modeling the behavior policy as a unimodal Gaussian distribution will limit its flexibility and rep- resentation ability. We consider capturing multi modes of the behavior policy. To that end, We
20

---

Published as a conference paper at ICLR 2023
Table 5: Averaged normalized scores on AntMaze five seeds. We compare IAC with IAC-MM, IAC-SR, IAC-VAE, and IAC-SNIS. Note that u=Umaze, u-d=Umaze- diverse, re-p-edium-replay, m-d-medium-diverse, 1-p-large-replay, and I-d-large- diverse.
Dataset
IAC-MM
IAC-SR
IAC-VAE
IAC-SNIS
IAC
antmaze-u-v2
52.7+2.9
72.1+5.0
84.0
40.0
77.6+
antmazo-u-d-v2
35.7
65.316.9
55.0=6.1
2.043.5
71.248.6
antmaze-m-p-v2
47.2=4.11
70.149.4
71.04444
78.0
72.0=7.6
antmaze-m-d-v2
1.6++0.3
65.2+t7.2
71.7=0066
72.0+7.8
74.2:t4.1
antmaze-l-p-v2
15.9+2.0
51.2++5.9
41.0+=1..5
47.3+10.0
57.0=7.44
antmaze-l-d-v2
6.3
40.1
51.3
30.0
47.219.4
antmaze-v2 total
159.8
364
374
269.3
399.2
experiment by modeling the action space discrete. Considering that the action range is [-1, 1], we split each action into 40 categories and each category has a range of 0.05. Then the behavior policy is estimated by cross-entropy. In this setting, the behavior policy is multi-modal.
We term this variant IAC-MM. We compare this variant and IAC and the result shown in Table and T The result shows in several settings the variant has marginally better performance. But the overall performance is worse compared to IAC. Especially in AntMaze tasks, IAC-MM suffers a performance drop. The reason might be that the classification task ignores the relation between nearby actions.
E DISCRETE DOMAIN
We also test IAC and IAC-w/o-B the CartPole task which has discrete action space. The dataset contains the samples in the replay buffer when we train a discrete SACCsoft-actoo--ritti)) until con- vergence. The result is shown in Fig. 7 IAC-w/o-3 has much worse final performance than IAC. Also, the maximum performance during training is worse than IAC.
IAC
IAC w/o
CartPole
Return
20
Epochs
Figure 7: Learning Curves of CartPole task.
F OTHER BENCHMARKS
To make comprehensive comparison, also compare AWAC and CRR with IAC. The results of MuJoCo locomation and AntMaze tasks are shown in Table 6 and Table respectively. The results show that methods have better performance than these baselines.
21

---

Published as a conference paper at ICLR 2023
Table 6: Averaged normalized scores on MuJoCo locomotion on five seeds. Other than the baselines above, we compare with AWAC and CRR. Note that m-medium, m-r=medium-replay, r=aadomm, m- e=medium-expert, and e-xxpen.
Dataset
BC
UneStep RL
ID3+BC
CQL
IQL
AWAC
CRR
IAC
hallcheetah-m--
42.011.7
50,4204
48.320.3
47.0:0..5
47,4:002
47,910.1
47.1:0011
51.610.3
hopper-m- m-v2
4..2.143
87.5.11099
59.3
23.0
66.2.15.7
59.8.0.77
38.1.1
14.6
walker2d-m-v2
71.066.5
84
83.7-2.1
73.1=170
78.3488.7
13.1
59.74:077
85.2::0.4
halfehhotal-n--44222
36.4+2.7
4..7+1.3
44.6=0..5
45.5+007
44.2412
44.8+0.11
44.4200.3
47.2+0.3
hopper-m-t-v2
21.8::055
98.54:2.77
6091188
18.7=12.9
94.718866
69.8
25.5:::1.6
103 103.2+::00
walker2d
61.7.116.3
73817.
78.11...6
helfehhotallmm-o-44
59.6:::.88
75.1
90.7=24.3
75.6=25.7
86.74.5.3
64.9422.3
85.2:1199
92.9=007
hopper m-e.v2
51.71:244
108.6:1:.11
98.01994
105.6:12.99
91.32:14.3
100.11:9.9
53.0.1..1
109 109.3::4.0
walker2d-nn e-v2
101.243366
111.3:0.44
110.1:0.5
1099116
10966:1:00
110.0110.2
9133114
110.110.11
halfehoctah-o-v2
92.9+005
88.2+6.55
9..7+1111
96.311
95.0::0.5
81.7444.2
93.5::0.77
94.5
hopper-e-v2
110:9:0033
106.9:4.1
96.5228.0
109.44005
109.5+155
108.7:+:.0
110.6119
walker2d-e-v2
107.7:10.11
110.7:20.4
110.2:0.3
108.5:10.5
10991:12
110.1::0.0
108.9:10.55
11481122
halfcheetah-
hopper-r-12
4.14:0.1
1.61::6
8.5
794004
7.940.2
9.24:0066
16.1
313300,3
walker20-r-v2
1,2.0.0
6.911.2
1.6
5.11133
5.411.2
0.2.10.7
4,9.0.88
3.0.1.33
locomation v2 total
784.2
1041.2
1013.2
1010.2
1033.1
975.6
817
1142,4
Table 7: Averaged normalized scores on AntMaze on five seeds. Other than the baselines above, we compare with AWAC and CRR. Note that u=Umaze, u-d=Umaze-divert m- m-p==meduum- replay, m-d=medium-diierrs. l-r=large-ryplay, and I-d-large-diveree.
OneStep
COL
IQL
antmazo-p-r2
66.846.7
54.0=3.4
13.0=:34.0
82.6#3.7
89.6:4-.2
80.0
43.8+2.3
77.6=3.8
antmaze-n-d-2
56.8:12.6
57.8:114.0
47.017.3
102266.7
65.6.18.3
52.016.9
42.8:1222
81.2-
antmano-m-p-v2
0.0:00.0
0.0:0.0
0.0:0.00
59.0+1.6
76.4+2777
0.0=0.0
0.44:00
72.0=7.6
aatmaze-m-d-2
0.0::000
0.6:20.5
0.2:0.44
46.61:24.0
728470
0.2:0..2
0.3:2022
74.2=4.1
antmaaee p-v2
0.0:0.00
antmaze-l-de 1.d-22
0.0:1:0.0
0.2:10.4
0.0:00.0
3.244.11
46.0.14.5
0.0.20.0
0.0:10.0
47.2-99.4
123.6
112.6
120.2
392.4
132.2
G OTHER ABLATIONS
G.1 ABLATION ON STATE RATIO
dF(s) might be helpful to IAC. For most RL settings, the dimen- sion of the state is larger than that of the action. Since high dimensional estimation is challenging,
Using state-distrin distrriittiittionn correction d=(+)
difficult to estimate d= (s) and ds). Thus we sort estimate
d (*)
d (s) by following the pa-
per Safinite-Horizon Off-Policy Estimation'. We term this variant IAC-SR. The result shown in
Table and Table 5 It indicates that introducing state-distribution correction worsens the perfor- mance. One reason is that the approximation for d' (x) not accurate and it will introduce bias to the algorithm.
da)
G.2 ABLATION ON SELF-NORMALIZED IMPORTANCE SAMPLING
Considering that self-normalized importance sampling also has lower variance than importance sampling. we test the performance of a variant with self- nerf-normalized importance sampling. This variant termed IAC-SNIS. The result of IAC-SNIS shown in Table and Table [5] The self- normalized importance sampling variant performs comparably to IAC MuJoCo tasks but per- forms worse than IAC on AntMaze tasks.
G.3 ABLATION ON Y
To study the hyperparameter 7 effect proposed method, We run the experiments of using different r in (0.1, 0.3, 0.5, 0.7, 0.9}. The result is shown in Fig. 8 It can be seen that the variant with large 7 performs better than with small 1)
22

---

Published as a conference paper at ICLR 2023
halfchectah-ranne randoom------dd-----
hopper-random--2 v2
30
40
IAC
IAC
Episode Retum
Episode Retum
15
20
0
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
n
n
antmaze- Laraazee-argg--aae large-
large-play-v2 play-v2
antmaze Larrizeee-rree-e-ee large--ireeerre diverse-v2 v2
80
80
IAC
IAC
Episode Retum
Episode Return
40
40
0.1
0.3
0.5 n
0.7
0.9
0.1
0.3
0.5
0.7
0.9
n
Figure 8: The effect of 17 on IAC.
Table 8: Runtime of TD3BC, CQL, IQL, IAC for haffcheetab-mcdduu--relllgg GeForce RTX 3090.
Algorithm Runtime
TD3BC
IQL
CQL
IAC
pre-training in IAC
1h
1h50min
4h10min
2h30min
2min
G.4 RUNTIME
We test the runtime of IAC on malfcheetah-medium--es IAC and other baselines are shown Table It takes 2h30min for IAC finish the task, which is comparable to other baselines. Note that it only takes two minutes for the pre-training part.
GeForce RTX 3090. The results of
23


Reviews:
[{"review_id": "Z2VvKIEaEo", "reviewer": "Reviewer_nmzD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the offline Reinforcement Learning setting, and more precisely the problem of computing Q-Values from transitions in the dataset, where bootstrapping from out-of-distribution actions (actions not in the dataset) may lead to over-estimation and poor learning. The proposed solution builds on sampling importance-resampling, and changes how the transitions used for learning are sampled from the dataset, to sample \"good\" (in-distribution) samples more often. These transitions are used to train a critic (using a somewhat conventional TD loss, with the main contribution being the sampling and not a new loss), and an actor, in the loss of which the estimated behavior policy of the dataset appears.\n\nExperimental results are thorough and show that the proposed method outperforms several baselines on challenging continuous-action tasks.", "summary_of_the_review": "The paper considers an important problem, proposes an elegant solution with a theoretical analysis, and empirically shows that the proposed method outperforms many baselines.", "strength_and_weaknesses": "Strengths:\n\n- The problem being considered is important and an active area of research. The proposed solution is elegant and outperforms the baselines. It also seems relatively easy to implement and computationally cheap (except that a SumTree needs to be added to the experience buffer)\n- The paper is clear and well-written, and theoretical details are given to motivate the approach, and justify the results.\n- Section 4.4, the practical implementation, is very well-written and concisely gives all the details needed for implementing the algorithm\n\nWeaknesses:\n\n- I did not find any big weakness of this work, only minor comments.\n- Starting from Equation 4, the notation mixes $\\Delta$ and $\\nabla$, without a (one-line) explanation of what these symbols mean. It is a bit weird to compute a TD-error from a gradient, so the equation looks like the $\\Delta$ is also a gradient (the gradient of the Q update, not the update itself). I find this quite confusing and explanations of the notations would be welcome.\n\nQuestion:\n\n- For offline RL papers, I'm always curious of what happens if the proposed algorithm is used in an online setting. In the case of this paper, is it possible to, every so often, let $\\pi_{\\theta}$ execute an episode in the environment, add the resulting transitions to the experience buffer, and continue learning from that?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EIxMwTr7gD4", "reviewer": "Reviewer_Ga1X", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes In-sample Actor Critic (IAC), a new algorithm for offline reinforcement learning. The main idea is to avoid extrapolation error by explicitly performing in-sample policy evaluation to learn the critic. To implement this, IAC applies sampling-importance resampling which has lower variance than a naive importance sampling implementation. The paper provides empirical studies in Mujoco locomotion control and AntMaze showing that IAC contains competitive performance compared to the state-of-the-art algorithms. ", "summary_of_the_review": "I recommend not accepting as the proposed algorithm might have large computational cost in practice and some claims are not well support by the experiment results. However, I am willing to adjust my score if the authors answer my questions in the rebuttal. ", "strength_and_weaknesses": "Strength\n\nAs discussed in the paper, offline RL algorithms suffer from extrapolation error by bootstrapping from out-of-distribution (OOD) actions. Previous methods mostly rely on using regularization to penalize OOD actions. But such methods are very sensitive to the regularization level. More importantly, when the data collection policy is very bad, the regularization actually prevents the algorithm from finding a good policy. What I really like about this paper is to rethink the problem from the first principle: is it possible to do bootstrapping from the in-sample actions? From this perspective I think this paper makes contribution to a very important problem in offline RL. The paper is also well organized and well written. Due to the high variance of importance sampling (IS), the paper proposes to use sampling-importance resampling (RIS). The authors then compare the variance of RIS and IS, and discuss how RIS is applied to the critic learning objective in offline RL algorithms. Most implementation details are clearly discussed in the paper. \n\n\n================================================\n\n\nWeakness: \n\nI have the following questions and comments. Please correct me if I missed anything important. I am happy to adjust my score based on how well the authors answer the questions in the rebuttal. \n\n1. My major concern is on the computation cost of the algorithm. It seems that after updating the current policy, IAC needs to adjust the weights (is ratio) for all (s,a) in the dataset.\n\n2. IAC needs to learn the behavior policy before training. How does it affect the performance if we train $\\beta$ as the other networks? Is there an ablation study for this? \n\n3. The paper suggests that one can simply treat $\\beta$ as a constant such that it’s canceled in the weights. In fact, from Table 1, we can see that w/o $\\beta$ even produce better results than original IAC. Is there an explanation on this? It’s always interesting to understand why something you try to approximate actually works better than the original algorithm. \n\n4. Following the previous question, I am wondering if it is because the domains are continuous control problems, where we can only observe one action in one state. Thus the log probability of $\\beta$ will be very close to zero. Then I am wondering how does IAC perform in discrete problems? \n\n5. IAC works very well empirically. But I am wondering why the algorithm performs so well, especially compared to IQL, which also performs in-sample bootstrapping. One limits of IQL in my opinion is that it’s hard to approximate the in-sample maximum with expectile regression when the behavior policy is very sub-optimal. But it seems to me that RIS also has the problem. I think it’s better to provide some evidence, even simple proof-of-concept experiments, to show the advantage of IAC over IQL. \n\n6. In Section 5.1, the authors suggest that IAC outperforms one-step RL as it has lower bias. But isn’t that the main advantage of IAC compared to one-step RL is to use multi-step DP?  (This doesn’t effect my score, I am just wondering if this discussion is important. )\n\n7. Finally, I am wondering what does OOD action exactly mean? The paper seems to suggest that OOD actions are those actions that are not in the dataset. Despite this is very reasonable, another view is to think of actions that are out of the support of the data collection policy. The difference is that for example an action can be covered by the behavior policy, but due to finite samples we don’t observe it in the dataset. It seems to be the second perspective is more interesting, as it allows generalization. Any comments on this? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E2GZvzfkPyg", "reviewer": "Reviewer_uBTY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents In-sample Actor-Critic (IAC), an algorithm for offline RL. The policy evaluation of IAC is done only using in-distribution samples of the dataset, where each sample's (normalized) importance ratio is used for resampling probabilities. Then, TD updates are performed using the resampled samples. For policy improvement, advantage-weighted regression is used to ensure that the learning policy samples actions within dataset support. The SIR is a consistent estimator and has a lower variance compared to the standard importance sampling. Experimental results show that IAC performs competitively with the state-of-the-art methods on Gym-Mujoco locomotion tasks and AntMaze tasks.\n", "summary_of_the_review": "I think this paper presents a sound offline RL algorithm of in-sample learning. My main concerns are technical novelty and some missing baselines in the experiments. ", "strength_and_weaknesses": "[Strengths]\n1. This paper presents an offline RL algorithm exploiting in-sample learning based on importance resampling, which enables stable learning yet achieves good empirical performance.\n\n\n[Weaknesses]\n1. It seems technical novelty is limited, given that IAC can be understood of a combination of Importance Resampling [1] for policy evaluation and AWAC[2]/CRR[3] for the policy improvement part.\n2. The algorithm requires behavior density estimation, which introduces additional computational costs and hyperparameters.\n3. In the experiments, comparison with some baselines is missing, e.g. AWAC and CRR[3], which are using similar approaches. Also, it would be great to compare with the method using self-normalized importance sampling instead of SIR.\n\n\n[Questions and comments]\n1. I am wondering if using state-distribution correction $d^\\pi(s) / d^\\mu(s)$ can further improve the performance.\n2. In addition to the OneStepRL and IQL, OptiDICE [4] only requires in-distribution samples $(s,a,r,s')$ of the offline dataset too.\n3. I am curious to see the result when using different $\\eta$ in Eq (12), e.g. $\\{0.1, 0.3, 0.5, 0.7, 0.9\\}$. Does $\\eta=1$ always perform the best?\n\n\n[1] Schlegel et al., Importance Resampling for Off-policy Prediction, 2019\n\n[2] Peng et al., Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019\n\n[3] Weng et al., Critic Regularized Regression, 2020\n\n[4] Lee et al., OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation, 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9kBrBs3N2q", "reviewer": "Reviewer_Y1J6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper applied the sampling-importance resampling methods to offline learning to conduct in-sample learning. The methods in this paper fit well into the line of in-sample offline RL learning.\n\n", "summary_of_the_review": "\n\nThe presentation of the paper is very clear and makes sense for in-sample offline learning. The theoretical analyses of the method are solid and well support the main idea of the paper. The experiments on various tasks are also sufficient to prove the effectiveness of the method.\n\nAlthough the method proposed in this paper is very interesting and contributes to the community, I cannot see why the superiority of the proposed method compared to existing other in-sample methods. \n\n\n\n", "strength_and_weaknesses": "\n\nStrength:\n\nThe method is very clear and makes sense for in-sample offline learning.\n\nThe paper contains several solid theoretical analyses of the method.\n\nThe experiments on Gym locomotion tasks and ANtMaze tasks are sufficient to show the effectiveness of the method.\n\nWeakness:\n\nOne limitation of the method is that it needs to know the probability of the in-sample action in the dataset. Although the paper also presented the result by removing the density estimator, this method does not perform well on AntMaze problem. Can you give the reason why the method does not perform well on the AntMaze-UMaze and AntMaze-Large datasets? The paper also didn't report the training time for using a density estimator.\n\nThe density estimator only uses Gaussian distribution to model the distribution. This often could limit the capacity as in practice the distribution of the dataset could hardly be a unimodal distribution.\n\n\n\nSeveral Questions:\n\nWhat's the training time for using a density estimator?\n\nI cannot see why the proposed method outperforms other in-sample offline RL methods. Could the author provide some insights on this? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster