RORL: Robust Offline Reinforcement Learning via Conservative Smoothing
Authors: Rui Yang
Chenjia Bai
Xiaoteng Ma
Zhaoran Wang
Chongjie Zhang
Lei Han
Venue: NeurIPS 2022 Accept (2022)
Keywords: Offline reinforcement learning
robust reinforcement learning
adversarial attack

Abstract: Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks. Due to the distribution shift issue, current offline RL algorithms are generally designed to be conservative in value estimation and action selection. However, such conservatism can impair the robustness of learned policies when encountering observation deviation under realistic conditions, such as sensor errors and adversarial attacks. To trade off robustness and conservatism, we propose Robust Offline Reinforcement Learning (RORL) with a novel conservative smoothing technique. In RORL, we explicitly introduce regularization on the policy and the value function for states near the dataset, as well as additional conservative value estimation on these states. Theoretically, we show RORL enjoys a tighter suboptimality bound than recent theoretical results in linear MDPs. We demonstrate that RORL can achieve state-of-the-art performance on the general offline RL benchmark and is considerably robust to adversarial observation perturbations.

RORL: Robust Offline Reinforcement Learning via Conservative Smoothing
Rui Yang'; Chenjia Bai?* Xiaoteng Ma², Zhaoran Wang', Chongjie Zhang?, Lei Han 5t
Hong Kong University of Science and Technology. *Shanghai AI Laboratory
Tsinghua University, "Northwestern University, Tencent Robotics
ryangam@connect. ust.uk .k, baichenjia@pjjab. org. ma ma-xx177maals...mmmmmmm tsinghua. edu. cn, zhaoranvangogaail. com chongjie@tsinghua. edu. en, leihan. cs@gmail.com
Abstract
Offiin reinforcement learning (RL) provides promising direction exploit massive amount of offline data for complex decision-making tasks. Due to the distribution shift issue, current offline RL algorithms are generally designed be conservative in value estimation and action selection. However, such conservatism can impair the robustness of learned policies when encountering observation devia- tion under realistic conditions, such as sensor errors and adversarial attacks. To trade off robustness and conservatism, we propose Robust Offline Reinforcement Learning (RORL) with novel conservative smoothing technique. In RORL, we explicitly introduce regularization on the policy and the value function for states near the dataset, as well additional conservative value estimation on these states. Theoretically, we show RORL enjoys a tighter suboptimality bound than recent theoretical results in linear MDPs. We demonstrate that RORL can achieve state- of-the-art performance on the general offline RL benchmark and is considerably robust to adveraaiill observation perturbations.
Introduction
Over the past few years, deep reinforcement learning (RL) has been a vital tool for various decision- making tasks [36] 49 47 M in trial-and-erro manner. A major limitation of current deep RL
algorithms that they require intense online interactions with the environment 30
These
data collecting processes can be costly and even prohibitive in many real-world scenarios such as
robotics and health care 30 53
Offline RL 14 28 gaining more attention recently since offers
probabilities to learn reinforced decision-making strategies from fully offline datasets.
The main challenge of offline RL the distribution shift between the offline dataset and the learned policy, which would lead to severe overestimation for the out-of-distributtin (OOD) actions [4] 28
To overcome such an issue, series of model-free free offline RL works 59
69
propose to celebrate conservatism. such as constraining the learned policy close to the supported distribution or penalizing the Q-values of OOD actions. Besides, another stream of works builds
upon model-basee algorithms [72] 71.58 58 which leverages the ensemble dynamics models to enforce pessimism through uncertainty penalizing or data generation.
However, conservatism is not the only concern when applying offline RL to the real world. Due to the sensor errors and model mismatch, the robustness of offline RL is also crucial under the realistic engineering conditions, which has not been well studied yet. In online RL, a series of works has been
"qqual Contribution Corresponding Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Without Smoothing
With Smoothing
Value
Value
Action
Action
State
Stase
Figure 1: A schematic diagram of smoothing in offline RL. The red spots represent the offline data samples. Without state smoothing, the value function would change drastically over neighboring states and induce an unstable policy. Yet, the smoothness may also lead to value overestimation of dangerous areas. RORL trades off' smoothness and possible overestimation as discussed in Sec
studied to learn the optimal policy under worst-case perturbations of the observation 73
20
environmental dynamics [57] 9 [ Yet, is non-trivial to apply online robust RL techniques into the offline problems. The main challenge is that the perturbation of states may bring OOD observation and extra overestimation for the value function. New techniques are needed to tackle the conservatism and robustness simultaneously in the offline RI..
This paper studies robust offline RL against adversarial observation perturbations, where the agent needs to learn the policy conservatively while handling the potential OOD observation with per- turbation. We first demonstrate that current value-based offline RL algorithms lack the necessary smoothness for the policy, which is visualized in Figure [ As illustration, baseline method CQL [29] learns a non-smooth value function, leading to significant performance degradation for even tiny scale perturbation on observation (see Section 3 for details). In addition, simply adopting the smoothing technique for existing methods may result in extra overestimation at the boundary of supported distribution and lead the agent toward unsafe areas.
show that famous
To this end, we propose Robust Offline Reinforcement Learning (RORL) with a novel conservative smoothing technique, which explicitly handles the overestimation of OOD state-action pairs. Specif- ically, we explicitly introduce smooth regularization on both the value functions and policies for states near the dataset support and conservatively estimate the values of these OOD states based on pessimistic bootstrapping. Furthermore, we theoretically prove that RORL yields a valid uncertainty quantifier in linear MDPs and enjoys tighter suboptimality bound than previous work [6].
In our experiments we demonstrate that RORL can achieve state-of-the-art (SOTA) performance in the D4RL. benchmark 12 with fewer ensemble Q networks than the current SOTA approach (2). The results of the benchmark experiments imply that robust training can lead to performance improvement in non- perturbed environments. Meanwhile, compared with current ensemble- considerably more robust to adversarial perturbations on observations. We conduct the adversarial experiments under different attack types, showing consistently superior performance several continuous control tasks.
baselines, RORL
2 Preliminaries
Offline RI. Considering an episodic MDP M = (S,
AS,,,,,, IP). where S is the state space, A is the action space, T is the length of an episode, r is the reward function, is the dynamics, and y is the discount factor. In offline RL, the objective of the agent is to find an optimal policy by sampling experiences from a fixed dataset D = {(s], at, ri, s Nevertheless, directly applying off-policy algorithms in offline RL suffers from the distribution shift problem. In Q-learning, the value function evaluated on the greedy action in Bellman operator TQ = + [max, (8', a')] tends to have extrapolation error since (s', a') has barely occurred in D.
Pessimistic Bootstrapping for Offline RL (PBRL) 6 an uncertainty-based method that uses bootstrapped Q-functions for uncertainty quantification
52 and OOD sampling for regularization.
Our code is available at http://gtthbb.com////gguugugg000

---

COL (action diff mixed order)
COL -smooth (action dift orderi
Score
140
sttack 0.00
attack c: 0.0
witack
attack
00
120
attack 0.14
242
wtlack 0.14
200
0.240
380
209
26-
COL
COL-
360
6.2
8.00
2.00
0
odz
obe
stare Oofter PGA
state Satter PCA)
ottacke
(a) Q Q-function of CQL
(b) Q-function of CQL-smooth
(c) Final performance
Figure 2: (a) (b) The Q-functions of & with adversarial noises in CQL and CQL-smooth, respectively. The same moving average factor is used in plotting both figures. (c) The performance of CQL. and CQI.-smooth with different perturbation scales. We use 100 uniformly distributed E [0.0, 0.15) for the evaluation.
Specifically, PBRL maintains K bootstrapped Q functions to quantify the epistemic uncertainty [5] and performs pessimistic update to penalize Q functions with large uncertainties. The uncertainty defined as the standard deviation among bootstrapped Q-functions. For each bootstrapped Q Q-function,
the Bellman target is defined TQ(s a) r(s, a) + p( s..)) 89) QQs', a') Au(s', a')] Under linear MDP assumptions, this uncertainty is equivalent to the LCB penalty and is provably efficient (24). Furthermore, PBRL incorporates OOD sampling by sampling OOD actions to form (s, andy pairs, where a and follows the learned policy. The detached learning target for (s, a ood) is
Frood ((s, dood) == a oood Xu(ss dood), which introduces uncertainty penalization to enforce pessimistic Q- Q-functions for OOD actions.
Smooth Regularized RL Robust RL aims to learn robust policy against the adversarial perturbed environment in online RL. SR 48 enforces smoothness both the policy and Q-functions. Specifically, SR encourages the outputs of the policy and value function to not change much when injecting small perturbations to the states. For state 8, SR constructs a perturbation set Ba(s. c) = (s d(ss, $) S t) with metric d(, ). which is chosen to be the lp distance, and introduces a smoothness regularizer for policy as RT Es-pp max;e3.(s.e) [[ is distance metric and the max operator gives an adversarial manner to choose 8. Similarly, the smoothness regularizer for the value function is defined as R. = Ee-p=
where D(-II-)
aa-a -a max (Q(ssa)
Q(5, a)) SR2L is shown to improve robustness against both random and adversarial perturbations.
3 Robustness of Offline RL: A Motivating Example
We give a motivating example to illustrate the robustness of the popular CQL 24 policies. We introduce an adversarial attack on state s to obtain & = arg max Dz(To (= where c)) = {8 d(s,3) () < <} is the perturbation set and the metric d(,) ) is chosen to be the foo norm. The Jeffrey's divergence Dj for two distributions P. Q is defined by: (P(l)) 1 (P]((( + (KI(Q]))) To obtain 8, we take gradient assent with respect to the loss function and restrict the outputs to the (s(s.)) set, where To is learned CQL policy. We remark that the the perturbation is applied on normalized observations following prior work ri
In the malker-medium-22 task from D4RL 12) we use various for adversarial attack to evaluate the robustness of CQL policics. Specifically, we use E E {0, 0.05, 0.1, 0.14} to control the strengths if 0. Given specific e, sample N state-action pairs
of the attack, where we have {(+,, };}} from the offline dataset, and then perform adversarial attack to obtain {(3, a,)} and the corresponding Q-values (5, ())), where the Q-function is the trained critic of CQL.
Figure 2(a) shows the relationship between & and the corresponding Qe with different e. To visualize S, we perform PCA dimensional reduction [55] and choose one of the reduced dimensions to represent 8;. More details can be found in Appendix B.3 With the increase of in the adversarial attack, the Q-curve has greater deviation compared to the curve with O. The result signifies that the Q-function of CQL not smooth in the state space, which makes the adversarial noises easily affect the Q values. As a comparison, apply the proposed conservative smoothing loss in CQL
3

---

Algorithm 1: RORL Algorithm
Bellman operation
Ensemble Soft Q-Learming
Initialize policy Ho and Q-functions (Qo.. Pox while not converged do
Sample mini-batch transitions a, a, 8') from D.
Offline Dusset
Perturbation
$, a
Conservative Smooth
Sample from Ba(s, é) to obtain (8, pairs.
Calculate the smooth loss Lemooth-
OOD
Sample OOD actions
Th (8))
sampling
Calculate uncertainty u(8, a) and the OOD loss Lood-
$,
Uncertainty Penalty
Train cach Q function Qo, with Eq. S
Train the policy with Eq.
Figure 3: RORL Algorithm: RORL trains multiple Q-functions for uncertainty quantification. The conservative smoothing loss is calculated for (8, a) with perturbed states. We perform uncertainty penalization for ($,a) with perturbed states and OOD actions.
training (i.e.. CQL-smooth) and use the same evaluation method to obtain á, and Qu. According to the result in Figure 2(b) the value function becomes smoother.
In addition, we show how the adversarial attack affects the final performance of offline RL policies. We use é E [0, 0.15] to evaluate both the original CQL policies (i.e., CQL) and CQL with conservative smoothing loss (i.e., CQL-smooth) in adversarial attack. Figure [ shows the performance with different settings of e. We find that our smooth constraints significantly improve the robustness of CQI,, especially for large adversarial noises.
4 Robust Offline RL via Conservative Smoothing
In RORL, we develop smooth regularization on both the policy and the value function for states near the dataset. The smooth constraints make the policy and the Q-functions robust observation perturbations. Nevertheless, the smoothness may also lead to value overestimation in areas outside the supported dataset. To address this problem, we adopt bootstrapped Q-functions [39 6 for uncer- tainty quantification and sample perturbed states and OOD actions for penalization. RORL obtains conservative and smooth value estimation on OOD states, which can improve the generalization ability of offline RL algorithms. The overall architecture of RORL is given in Figure t
Robust Q-function We sample three sets of state-action pairs and apply different loss functions to obtain a conservative and smooth policy. Specifically, for (s, a) pair sampled from D, we construct a perturbation (s, e) obtain (s, a) pairs. where e (s,s c) and is the perturbation scale. The perturbation set Ba(s, c) {& d(s. 8) I c) for state is an 6-radius ball measured in metric d(.). which is the & norm in our paper. Then we perform OOD sampling by using the current policy Ty to obtain ($, â) pairs, where à ~ (8). RORL contains K ensemble Q-functions. We denote the parameters of the i-th Q-function and the target Q-function as Φ; and o/,, respectively. In the following, we give different learning targets for (s,a), (8, a), and (8, a) pairs.
First, for (s, a) pair sampled from D, we apply extended soft Q-learning to obtain the target as TQ, == r(s, a) + ( j=I...,
min
(s' a') log (e(a'ss')),
(I)
where the next-Q function takes minimum value among the target Q-functions and log (o (s') the entropy regularization. Note that Eq. is the same learning target of SAC-N in
is
Then, for (8, a) pair with a perturbed state, we enforce smoothness in each Q-function by minimizing the Q-value difference between Q(s, a) and Q(8, a). In particular, we choose an adversarial Ba(s, that maximizes inner objective L(Q(S, a), Q(s, a(). and then train each Q-function to minimize a loss function Lamooth with the adversarial 3. Intuitively, we want the Q-function to be smooth under the most difficult (i.c., adversarial) perturbation in Ba(s, e). The smooth loss function for Qo, is as follows:
Samooth (smoott(s,,
max (S. Q). (E.e(a.e)
We denote 8(8, 8, a) = (6, ()) Qa, (s, a) and remark that if 8s,, a) > O, the perturbed state may induce overestimated Q-value that we need to smooth. In contrast, if ö(s, 6, a) 0, the

---

perturbed Q-function is underestimated, which does not cause a serious problem in offline RL. As result, we use different weights for ö(s, 8, a)+ and 0(s, a)_, where I4 max(r, 0) and x- min(ar, 0). The definition of ((::-) is give as follows:
(l de ($, a), Qs (s. a)) = (1 T), 2) _ + rō(s, 8)
(3)
where we can choose T S 0.5. Ssmooth; we does not introduce OOD action for smoothing since the actions are desired to be close to the behavior actions for areas near the offline dataset.
Finally, to prevent overestimation of OOD states and actions, use bootstrapped uncertainty u($, a) as the penalty for Q(s, â). where â ~ TT(()) is an OOD action sampled from the current policy we- We remark that similar OOD sampling is also used in PBRL [6]. The difference is that PBRL only penalizes the OOD actions for in-distribution states, while RORL penalizes both the OOD states and OOD actions to provide conservatism for unfamiliar areas. We follow PBRL and use loss function as:
(8, Lood (s; Φi) = Es--B(ssr) a== (3. Qc.
(4)
where the pseudo-target for the OOD datapoints is computed as: ToodQo. u(8, a), which is detached from gradients similar to the conventional TD target. The bootstrapped uncertainty u(s, a) is defined as the standard deviation among the Q-ensemble:
o. (8, a) (s,
u(s,a) a) = V Σ k=1 (s,, (o))))))
K
The ensemble technique 33 forms an estimation of the Q-posterior, which yields diverse predictions and large penalty u(8, â) on with scarce data.
Combining the loss functions above, RORL has the following loss function for each Qo.:
min E. ann (TQs, (s, (s. + Sa (s, a; Φ.) + Food (s; 0x)
(5)
Robust Policy We learn a robust policy by using smooth constraint to make the policy change less under perturbations. Similarly, we choose adversarial state & Ba(s,e) that maximizes (()T((1s)] and then minimize the policy difference between (e(sss) and ne(-1S). To conclude, we minimize the following loss function for nt:
min (8 + log ma(as)) + 3r max Dy (T(To(-1s)| ]: j=I..... K
min (a-DDortollel---
sEBals.c)
where the first term aims to maximize the minimum of the ensemble Q-functions to obtain (6) conservative policy, and the second term is the entropy regularization.
5 Theoretical Analysis
We analyze a simplified learning objective of RORL in linear MDPs 23
24. where the feature map of the state-action pair takes the form of S x A - Rd, and both the transition function and the reward function are assumed to be linear ø. The parameter W, of RORL can be solved in closed form following the least squares value iteration (LSVI), which minimizes the following loss function.
w
min w Rd
Σ
(ut (s)s),aa))² +
Σ
Σ
(Qw(s), (w, 2+)
Balst e)
Σ (w
(i (um(((()))))
(7) where we have Qu at) at) i since the Q-function is also linear in Φ. The first term in Eq. is the ordinary TD-error, where we consider the setting of 1 and the Q-target vi r(s), at) + ((41(5)41). The second term is the proposed conservative smoothing loss. Specifically,, 8t ~ (s))(ss)) are sampled from a T ball of center sf and norm e > 0, which can also be formulated as 8f Ba(st, e). The third term is the additional OOD OOD-sampling loss, which enforces
5

---

conservatism for OOD states and OOD actions. In contrast to PBRL 6 we use perturbed states
sampled from ood D (s) rather than states from dataset. The OOD action is sampled from policy The explicit solution of Eq.
7 takes the following form:
A
Σ
Ms ajvl +
Σ
((3,
(8)
(B,a.8)m~Dan
where the covariance matrix Ag is defined as
172
A
( a) a))((s)) a))
E
o(s,, (,);;,))
(3,0)~Danl
(9)
Σ
[0($) - o(s) a.l,:,)][[(8))..oo a3) -6))
Ba ( e)
We denote the first term and the second term A and Apod, which represent the covariance matrices induced by the offline samples and OOD samples, respectively. Nevertheless, in linear MDPs, is difficult to ensure the covariance A Agod 2 A-1, since it requires that the embeddings of the samples isotropic make the eigenvalues of the corresponding covariance matrix lower bounded. This condition holds if we can sample embeddings uniformly from the whole embedding space. However, since the offline dataset has limited coverage in the state-action space and the OOD samples come from limited lo -balls around the offline data, X + Ag cannot be guaranteed to be positive definite. PBRL [6] uses the assumption of Ar 2 A-1, while it is unachievable empirically. In RORL, we solve this problem by introducing an additional conservative smoothing loss, which induces a covariance matrix as Ag godd_dif
Σ [a(stee) ExpDDan(tt) (0(s), at) (st (o(),, at) (sf, at)] (i.e.,
the third term in Eq. The following theorem gives the guarantees of Aqua 2 1 Theorem 1. Assume
E [1, m) the vector group of all it ~ Dooddss): {0(s, at) s5st, at} be full rank, then the covariance matrix Ag Aood_diff diff is positive-definite: A; doif 2 A where X > 0.
Recall the covariance matrix of PBRL is ÄPBRI. = -imm + Ägnd, and RORL has covariance matrix as A, APBRL + A diff have the following corollary based on Theorem
Corollary Under the linear MDP assumptions and conditions Theorem we have A. 2 A APBRL,
Further, the covariance matrix &s of RORL is positive-definite: A₁ A- where X >> 0.
Recent theoretical analysis shows that an appropriate uncertainty quantification is essential provable efficiency in offline RÍ. [24 65 6. Pessimistic Value Iteration (24 defines a general E-uncertainty
quantifier as the penalty and achieves provable efficient pessimism in offline RL. In linear MDPs, Lower Confidence Bound (LCB)ypenalty II 23 is known to be a E-uncertainty quantifier for
appropriately selected 3, as pleb ax) = Br [db(s,, a.) (A- a.)] 1/2 Following the analysis of PBRL. G since the bootstrapped uncertainty is an estimation of the LCB-penalty and the OOD
sampling provides covariance matrix A, ż= A given in Corollary 1 the proposed RORL also forms a valid É-uncertainty quantifier. This allows us to further characterize the optimality gap based on the pessimistic value iteration 24 We have the following suboptimality gap under linear MDP assumptions.
Corollary 2. SubOpt(#*, #) < 1= E,- [Ilc a:)]
T Or=1
E.- [rkcb_ PBRL,
(st, at)].
Detailed proof can be found in Appendix suboptimality bound than PBRL 6
A Corollary indicates that RORL enjoys a tighter
Experiments
We evaluate our method on the D4RL. benchmark 12 with various continuous-control tasks and
datascts. We compare RORL with several offline RL algorithms, including (i) BC that performs

---

Table Normalized average returns on Gym tasks, averaged over random seeds. Part of the results are reported in the EDAC paper. Top two scores for each task are highlighted.
SAC-10
EDAC
EDAC-10 (Reproduced)
RORL (Ours)
Task Name
BC
CQL
PBRL
(Reproduced)
(Paper)
halfchoctab-randde
2.2+0.0
3133+355
11.055.8
19,0=1.5
28.4=100
134 1.1
28.5:+0.8
hulfcheetah-meelee medium
43.21:0.6
46,9104
$7.915
64,9=1.3
65.9:0.6
64.1=1.1
66.8:10.7
halfcheetah medium expert
44.01.16
95.0114
92.31111
107.1-200
1.6.3.1.9
107.2.1.0
107.8:1.1
37.6+22.1
45.3+-0.3
45.148.0
63.2-0.6
61.3+119
60.1++0.3
61.9+11.5
halfchoctab-cxpert
91.8+1.5
97.3+11
92.41777
104.9=099
106.8=34
104.0=0.88
105.2:20.7
hopper-randee random hopper medium
3.71:0.6
5.310.66
26.819.3
25.9-9.6
25.3:1:0.4
16.9110.1
31,4:10,1
54.1
61,9164
75.3
0.8-0.2
1.
103.6:10.2
104.8 10.1
wopper-medium-exppe exppet
53.94-477
96,9+15.1
110.8+0.8
6.1-77
110.7=0.1
58.14-22.3
112.7+0.2
hoper--nciimm-cplay
16.6+2.88
86,347.33
100.6#10
10299=0.9
101.0=0.5
102.8:0.33
102.8=0.55
hopper-expert
107.719.7
106.549.1
110.5:0.
1.1=0.5
110.1
77.01-3.9
11288:0.2
walker2d-random
1.320.11
5.41177
8.14-444
1,5111
16.6=7.0
6.748.8
21,4:10,2
walker2d-mediee
70.911110
79,51:3.2
89.6:10.77
46.7:145.3
9..5
87.611110
102.4:11.4
walker2d medium expert
116.7
114.710.9
115.4:10.5
121.2
mulker2d-medium-ee
20.3++9..
76.8+ 10.0
17.7+14.5
89.6=3.1
87.1+2.3
94,0+112
90.4 0.5
walker2d-cxpert
108.7+0.2
109.340.11
108. 10..3-10.3
1.2=0.7
115.1=199
57.8455.7
115.4
Average
50.8
82.9
85.7
Total
746.1
1052.8
1116.5
761.6
1243.4
1068.7
1285.7
behavior cloning, (ii) CQL
29 that learns conservative value function for OOD actions, (iii) EDAC 2 that learns a diversified Qensemble to enforce conservatism, and (iv) PBRL [6] that performs uncertainty penalization and OOD sampling. We also include basic SAC- algorithm as baseline
21. which is an extension of SAC with 10 Q-functions. Among these methods, EDAC [2] and PBRL 6 are related to RORL since all these methods apply Qeensemble for conservatism. EDAC needs much more Q-networks (i.e., 10~50) for hopper tasks than PBRL and RORL that only use 10 Q-networks. For fair comparison, we also report the reproduced results of EDAC-10. To assign uniform adversarial attack budget each dimension of observations, for SAC-10, 10, EDAC and RORL. Besides, we use different perturbation scales for the policy smoothing loss, the Q smoothing loss and the OOD loss, namely ép, EQ and food More ppram-prraetters and implementation details are provided in Appendix
normalize the observations
6.1 Benchmark Results
We evaluate each method on Gym domain that includes three environments (HalfCheetah, Hopper, and Walker2d) with five types of datasets (random, medium, medium-replay. medium-expert, and expert) for each environment. The medium-replay dataset contains experiences collected in training medium- level policy. The mandonn/medduum/epper dataset is generated by a single random//medium/epper policy. The medium-expert dataset is a mixture of medium and expert datasets. For benchmark experiments, we set small perturbation scales ép, EQ. and food within {0.001, 0.005, 0.01} when training RORL and do not include observation perturbation in the testing time.
Table reports the performance of the average normalized score with standard deviation. (i) SAC-10 is unstable on several walker2d and hopper tasks since the ensemble number is relatively small to provide reliable uncertainties for SAC-N (2). (ii) EDAC solves this problem by gradient diversity constraints while still requiring 10~50 Q-networks to obtain reasonable performance. In contrast, RORL only uses 10 ensemble nentwwrks to achieve better or comparable performance with EDAC. Additionally, we also show that RORL outperforms EDAC-10 by large margin. (iii) PBRL chooses an alternative OOD-sampling technique to reduce the ensemble numbers. According to the result, RORL significantly outperforms PBRL with the same ensemble number. The reason is RORL additionally uses conservative smoothing loss for perturbed states and penalizes values of these states based on uncertainty estimation. which may improve the generalization ability of the learned policy on continuous state space. We remark that RORL significantly improves over the current SOTA results on walker2d and hopper tasks, probably because these two tasks require a more precise balance of conservatism and robustness for better performance.
6.2 Adversarial Attack
We adopt three attack methods, namely random, action diff. and min Q following prior works A Given perturbation scale e, the later two methods perform adversarial perturbation observations
7

---

ROAL
SAC-10
halfcheedal-medimm-2
hallcheetah-mediumee action diff
bollcheeta--mediumnae action diff order
hallcheetah-meddam--2
hallcneetal-meddims
mixed
40-
40
40
20-
20
0.0
0.1
0.2
6.3
0.0
0.1
0.2
0.3
0.0
011
0.
0.3
0.0
6.1
0.
0.3
0.0
0.1
0.2
6.3
Sc3lc
scale
2calc
scale
scals
(a) Performance under attack on the halfchhctah-mmcdiume dataset
RDAL
SAC-10
walker2d medium-vi
walkerr2d -i---mmmm22
walker2d waller2d-mediuum-ee
walker2d-mediune
diff
1.0
00-
80
50
60-
60
60
50
20
20
0.0
0.1
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.0
0.1
0.2
0.3
acale
acale
scale
(b) Performance under attack the
dataset
RDAI
EDMC
SAC-15
hopper-ediumm-vV
hooper-medium-yg
happer-mediume
hooper-medium-vz
hopper-medlurg mixed
Q
1.0
100
1000
100
80-
80
60-
60-
40-
40
40
20
0.0
0.2
0.1
0.2
0.0
0.5
0.0
0.1
0.2
0.3
Scale
Scale
scale
(c)
Performance under attack on the hopper-medium--2 dataset
Figure 4: (a) (b) (c) illustrate the performance of RORL, EDAC and SAC-10 under attack scales range [0, 0.3) of different attack types. The curves are averaged over seeds and smoothed with window size of 3. The shaded region represents half a standard deviation.
and are given access to the agent's s policy and value functions. Details about the three attack methods follows.
random uniformly samples perturbed states in an Loo ball of norm e.
action diff' is an effective attack based on the agent's policy and is proved to be an upper bound on the performance difference between perturbed and unperturbed envi- ronments [73]. It directly finds perturbed states in an Lo0 ball of norm to satisfy: max;e Da ([)((- i.c., minlse -D (- ([)(
min Q requires both the agent's policy and value function to perform a relatively stronger attack. The attacker finds perturbed state minimize the expected return of taking an action from that state: min ((s, aa (3)). For ensemble- based algorithms, Q is set the mean of ensemble Q functions.
In our experiments, the two objectives of action diff and min Q are optimized via two ways. Specifi- cally, we optimize the objectives through:
selecting the best perturbed state from uniformly sampled 50 states, which has the advantage of simplicity and little computation cost. For attacks with this type of optimization, we use their original names without specifying.
uniformly sampling 20 initial states and performing gradient decent for 10 steps with step size of t from each initial state to find the best perturbed state. Note that need to clip

---

the perturbed states within the loe ball at the end of each optimization step. Among the attacks using this optimization, we specifically remark "mixed-order" in their names.
We compare RORL with ensemble-based baselines EDAC and SAC-I0 on halfcheetahh-mediun--Z walker2d-medium-222 and hopper-medium-Ve datasets. To handle large adversarial noise, we set the perturbation scales ép. EQ and Good within {0.005, 0.02, 0.03, 0.05, 0.07) in RORL's training phase. More detailed description the results, RORL exhibits improved robustness than other baselines under five types of adversarial attacks. On the other hand, we find that random attack is not effective for ensemble-basee offline RL algorithms, and the "mixed order" attack brings more significant performance drop than vanilla zero-order optimization.
be found in Appendix B The results are shown in Figure g In
RGAL
random
action diff
action diff mixed order
min Q
min Q mixed order
80
det srore
60-
60
60
60
60
20
20
20
20-
20-
0.0
0.1
0.2
0.0
0.1
0.2
0.1
0.2
0.1
0.2
0.0
0.2
0.3
Scale
scale
Scale
scale
Scale
Figure 5: Ablation studies on the walker2d-medium-ye v2 dataset with varying perturbation scale. The curve averaged across random seeds and smoothed with window size of The shaded region represents half a standard deviation.
6.3 Ablations
We conduct ablation studies on the walker26-medium--2 dataset to evaluate the importance of three terms, i.e., the policy smoothing loss, the Q smoothing term and the OOD loss. From the results in Figure 5 we can conclude that each loss contributes to the performance of RORL under adversarial observation attacks. The OOD loss is the most essential term, without which the performance is worse than RORL at almost all perturbation scales and all types of attacks. The policy smoothing loss also important, especially for perturbation scales larger than 0.2. addition, Q smooth loss has the minimal impact, which is reasonable since the basic algorithm SAC-10 is based on 10 ensemble Q networks. More ablations on the number of Q networks, the effect of food and r, and a comparison with more baselines can be found in Appendix
6.4 Computational Cost Comparison
We compare the computational cost of RORL with prior works on a single machine with one GPU (Tesla V100 32G). For each method, measure the average epoch time (i.e., xx 10 training steps) and the GPU memory usage the hopper-medium-Ve task. More discussions are provided in Appendix C.T
Table 2: Computational costs.
Runtime (slepoch)
GPU Memory
(GB)
CQL
32.40
1.4
SAC-10
12.73
1.3
As shown in Table 2 RORL runs slightly faster than CQL and much faster than PBRL. PBRL is so slow because it uses 10 Q networks and needs OOD action sampling. In RORL, we also include the OOD state-action sampling and the robust training procedure, but implemented these procedures efficiently based the parallelization of Q networks. Even so, RORL still slower than SAC-10 and EDAC. As demon- strated in our experiments, RORL enjoys significantly better robustness than EDAC and SAC-10 under adversarial perturbations. Regarding the GPU memory consumption, RORL uses comparable memory to PBRL and EDAC, with only 16.7% 7% more memory usage.
PBRL
102.96
1.8
EDAC
17.94
1.8
RORL
29.56
2.1
7 Related Works
Offline RI Research related to offline RI. has experienced explosive growth in recent years. In model-free domain, offline RL methods focus on correcting the extrapolation error [4] in the off-

---

policy algorithms.
The natural idea is to regularize the learned policy near the dataset distribution 159
66 For example, MARVIL reweights the policy with exponential advantage. which implicitly guarantees the policy within the -I--divegeence neighborhood of the behavior policy. Another stream of model-free methods prevents the selection of OOD actions by penalizing their
63
37
61
69
13
With the ensemble Q networks and the additional loss achieves SOTA performance in the D4RL benchmark. proposes an explicit value underestimation of OOD actions according to the uncertainty, which requires fewer ensemble networks. Inspired by EDAC and PBRL, we build our work upon ensemble networks, focusing more on the smoothness over the state space.
Q-value 28 29 2 I0O or V-learning
K%
27
term to diversify their gradients, EDAC
2
Instead of diversifying gradients, PBRL
6
Besides the surprising empirical results. theoretical analysis of offline reinforcement learning algo- rithms of increasing interest 9 24
7 Though the assumptions for the dataset vary the different papers, they all suggest that pessimism and conservatism are necessary for offline RI. Our
45
555
theoretical results can be viewed as robust extensions to previous theoretical results 24
Robust RL The research line of robust RL can be traced back to -contrl theory where policies are optimized to be well-performed in the worst possible deterministic environment. Depending on the definition, there are different streams of research on robust RI. As the extension of robust control to MDPs, Robust MDPs (RMDPs) 38 21
are proposed to formulate the perturbation of transition probabilities for MDPs. Though some recent analyses with theoretical guarantees come out under specific assumptions for RMDPs 775
46
[9]
68 M. there is currently no
practical algorithm to solve RMDPs in a large-scale problem, expect some linear approximation attempt [.]. In online RL, domain randomization [50 B5 assumes the model uncertainty can be predefined data collection by changing the setup of a simulator. However, not practical for offline RL. Robust Adversarial Reinforcement Learning (RARL) 4 and Noisy Robust Markov Decision Process (NR-MDP) 25 study the robust RL with the perturbed actions, showing that the policy robustness to adversarial or noisy actions can also induce robustness for model parameter changes. The most related work to ours is SR significant performance improvement in the online setting. In contrast, we focus on the offline setting and tackle the potential overestimation of perturbed states. Another related work is S4RL [50], where the authors study different data augmentation methods to smooth observations in offline RL. Their result supports the necessity of state smoothing. More related works are discussed in Appendix E
48 which shows policy smoothing can lead to
8 Conclusion
We propose Robust Offline Reinforcement Learning (RORL) to trade-off conservatism and robustness for offline RL. To achieve that, we introduce the conservative smoothing technique for the perturbed states while actively underestimating their values based on pessimistic bootstrapping to keep conser- vative. We show that RORL can achieve comparable or even better performance with fewer ensemble Q networks than previous methods in the offline RL benchmark. In addition, we demonstrate that RORL is considerably robust to adversarial perturbations across different types of attacks. We hope our work can promote the application of offline RL. under real-world engineering conditions.
The main limitation of our method is that the adversarial state sampling slows down the computing process, which may be improved in future work. Also, an interesting direction is to smooth or penalize the policy and Q functions in latent spaces rather than the normalized observation space.
Acknowledgements
This work was in part supported by Tencent Robotics X and Shanghai AI Laboratory, and in part by Science and Technology Innovation 2030- "New Generation Artificial Intelligence" Major Project (No. 201 8AAA01009004 and National Natural Science Foundation of China (621 (62176135). The authors would like to thank the anonymous reviewers. Rui Yang thanks Yi Wang and Haoyi Song for valuable discussion.
10

---

References
[1] Yasin Abbasi- Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in neural information processing systems, volume 24, pages 2312-2320, 2011.
[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uneertainty- offline reinforcement learning with diversified g-ensemble. Advances Neural Information Processing Systems, 34, 2021.
[3] Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, 2017.
[4] Chenjia Bai, Lingxiao Wang, Lei Han, Animesh Garg, Jianye Hao, Peng Liu, and Zhaoran Wang. Dynamic bottleneck for robust self- supervide exploration. Advances in Neural Information Processing Systems, 34:17007-17020 2021.
[5] Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, and Zhaoran Wang. Principled exploration via optimistic bootstrapping and backward induction. In International Conference Machine Learning, pages 577-587. PMLR, 2021.
[6] Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driver offline reinforcement learning. In International Conference on Learning Representations, 2022.
[7] Tamer Başar and Pierre Bernhard. H-infinity optimal control and related minimax design problems: a dynamic gume approach. Springer Science & Business Media, 2008.
[8] Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induction attacks. In International Conference on Machine Learning and Data Mining in Pattern Recognition, pages 262-275. Springer, 2017.
[9] Jinglin Chen and Nan Jiang. Information-theerrtic considerations in batch reinforcement learning. In International Conference Machine Learning, pages 1042-1051. PMLR, 2019,
[10] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. ar preprint arXiv:2202.024444 2022.
[11] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7844):580-558) 2021.
[12] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[13] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[14] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML, 2019.
[15] Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. arXiv preprint arXiv:2205. 33703 2022.
[16] Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representations, 2019.
[17] Florin Gogianu, Tudor Berariu, Mihaela C Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference on Machine Learning, pages 3734-3744. PMLR, 2021.
[18) Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint 1rX.v 2014.

---

[19] Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast Bellman Updates for Robust MDPs. In Proceedings of the 35th International Conference on Machine Learning, pages 1979-1988. PMLR, 2018.
[20] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprini arXiv: 1.Xi::7702.02244 2017.
[21] Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-280, 2005.
[22] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34, 2021.
[23] Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137- 2143. PMLR, 2020.
[24] Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International Conference on Machine Learning, pages 5084-5096. PMLR, 2021.
(25) Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher. Robust reinforcement learning via adversarial training with langevin dynamics. Advances in Neural Information Processing Systems, 33:8 13:8127-8138, 2020.
[26] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In NeurIPS, 2020.
[27] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2021.
[28] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In NeurlPS, 2019.
[29] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurlPS, 2020.
[30] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643 2020.
[31] Jialian Li, Tongzheng Ren, Dong Yan, Hang Su, and Jun Zhu. Policy learning for robust markov decision process with a mismatched generative model. arXiv preprint arXiv:2203.06588 2022.
[32] Lanqing Li, Rui Yang, and Dijun Luo. Focal: Efficient fully-offline meta-reinforcemmnnt learning via distance metric learning and behavior regularization. In International Conference Learning Representations, 2021.
[33] Xiaoteng Ma Yiqin Yang, Hao Hu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, Bin Liang, and Qihan Liu. Offline reinforeement learning with value-based episodic memory. In International Conference on Learning Representations, 2022.
[34] Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Jerry Zhu. Policy poisoning in batch reinforcement learning and control. Advances in Neural Information Processing Systems, 32, 2019.
[35] Bhairav Mehta, Manfred Diaz. Florian Golemo, Christopher Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning, pages 1162-1176 1166 PMLR, 2020.
[36] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 518(7540):5299553 -333, 2015.
[37] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.093: 09359, 2020.
[38] Arnab Nilim and Laurent Ghaoui. Robustness in markov decision problems with uncertain transition matrices. Advances in neural information processing systems, 16, 2003,
12

---

[39] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In NeurlPS, 2016.
[40] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Anan- thram Swami. Practical black-box attacks against deep learning systems using adversarial examples. arXiv preprint arXiv: 1502.026077 1())3, 2016.
[41] Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary, Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03666 2017.
[42] Anay Pattanaik, Zhenyi Tang. Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2040-2042, 2018.
[43] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817-2826.
PMLR, 2017.
[44] Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learn- ing robust neural network policies using model ensembles. arXiv preprint 1610.01610..11: 2016.
[45] Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34, 2021.
[46] Aurko Roy, Huan Xu, and Sebastian Pokutta. Reinforcement learning under model mismatch. Advances in neural information processing systems, 30, 2017.
[47] Julian Schritwwesser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(783)):604-66)) 2020.
[48] Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pages 8707-8718. PMLR, 2020.
[49] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529((587):484 489, 2016.
[50] Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4rl: Surprisingly simple self- for offline reinforcement learning in robotics. In Conference on Robot Learning, pages 907-917. PMLR, 2022.
[51] Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. Exploiting reward shifting in value-based deep rl. In Advances in Neural Information Processing Systems, 2022.
[52] Hao Sun, Boris van Breugel, Jonathan Crabbe, Nabeel Seedat, and Mihaela van der Schaar. Daux: density-based approach for uncertainty explanations. arXiv preprint arXiv:2207.05161, 2022.
[53] Hao Sun, Ziping Xu, Meng Fang, Zhenghao Peng, Jiadong Guo, Bo Dai, and Bolei Zhou. Safe exploration by solving early terminated mdp. arXiv preprint MXXv:2107.04200. 2021.
[54] Aviv Tamar, Huan Xu, and Shie Mannor. Scaling up robust mdps by reinforcement learning. arXiv preprint arXiv:/366.61 333066689, 2013.
[55] Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series (Statistical Methodology). 61()))611 1-622, 1999.
13

---

[56] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In 2017 IEFF/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[57] Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008. 01825, 2020.
[58] Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. Advances in Neural Information Processing Systems, 34, 2021.
[59] Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018.
[60] Ruosong Wang, Simon S Du, Lin F Yang, and Ruslan Salakhutdinov. On reward-free rein- forcement learning with linear function approximation. In Advunces in neural information processing systems, 2020.
[61] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768- -778,, 2020.
[62] Fan Wu, Linyi Li, Chejian Xu, Huan Zhang, Bhavya Kailkhura, Krishnaram Kenthapadi, Ding Zhao, and Bo Li. Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks. arXiv preprint arXiv:2203.0883 2022.
[63] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 191..119111,,,,,,,11111111111111 2019.
[64] Lihua Xie and Carlos E de Souza. Robust h/sub infinity/control for linear systems with norm- bounded time-varying uncertainty. In 29th IEEE Conference on Decision and Control, pages 1034 1035. IEEE, 1990.
[65] Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman- consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34, 2021.
[66] Rui Yang. Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. In International Conference on Learning Representations, 2022.
[67] Tianpei Yang, Hongyao Tang. Chenjia Bai, Jinyi Liu, Jianye Hao, Zhaopeng Meng, and Peng Liu. Exploration deep reinforcement learning: comprehensive survey. arXiv preprint arXiv:2109.06668, 2021.
[68] Wenhao Yang, Liangyu Zhang, and Zhihua Zhang. Towards theoretical understandings of robust markov decision processes: Sample complexity and asymptotics. arXiv preprint 2055v22/05.03863, 2021.
[69] Yiqin Yang, Xiaoteng Ma, Li Chenghao, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[70] Ming Yin, Yaqi Duan, Mengdi Wang, and Yu-Xiang Wang. Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. arXiv preprint arXiv:2203.0558
2022.
[71] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34, 2021.
14

---

[72] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In NeurIPS, 2020.
[73] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho- Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations state observations. Advances in Neural Information Processing Systems, 33:21024 1022102221127 2020.
[74] Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Robust policy gradient against strong data corruption. In International Conference Machine Learning, pages 12301-12401. PMLR, 2021.
[75] Zhengqing Zhou, Zhengyuan Zhou, Qinxun Bai, Linhai Qiu, Jose Blanchet, and Peter Glynn. simite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages
PMLR,
2021.
15

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See See I
and
Appendix B
(b) Did you specify all the training details (e.g., data splits, hyper- hyper- parameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e-g., with respect the random seed after running experi-
ments multiple times)? [Yes] See See
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix B
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cited D4RL
and EDAC for their datasets and code.
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We included our code in the anonymized link.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] Opensource code and dataset.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "Y3RKy0EuzBc", "reviewer": "Reviewer_KVko", "summary": "The authors propose an approach to offline reinforcement learning that is robust to small perturbations in the observation space such that the changes are not detrimental to the performance of the final policy. The achieve this by encouraging the value estimator network to be smooth over the state space while being conservative on out of distribution samples. Moreover the learned policy is also constrained to change less with these perturbations.  Experimental results show that the proposed algorithm is able to perform competitively with current Robust / Baseline approaches, and enjoy increased robustness over adversarial attacks. ", "questions": "Q1: I could not find a definition for the distance metric d used to calculate the epsilon (line 90). Is this a L2 distance metric ?\nQ2: Can the epsilon be normalized so that it is takes into account the overall variation of the state space as well as the dimensionality. ?  \nQ3: What parts are highlighted in the table 1, is this discounting the scores that are not statistically significant during ranking? \nQ4: Should the score for scale 0 attack at figure 4(b) be same for all attacks and match the evaluation score at table 1? If not what is the effect of scale zero on the final policy. ", "limitations": "While the authors clearly mention the adversarial state sampling as their main limitation, they do not quantify this slow down for different state sampling approaches. It would be interesting to see the actual effect on the compute time as a percentage of the total training time.\n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nS1: Overall the motivations for the approach are intuitive and easy to follow. The paper is well written and clear. (excluding some minor comments in the questions section.) \n\nS2: The proposed attack metrics are diverse and well defined under the current framework. The experimental results show that RORL is more robust towards the proposed attacks as compared to other methods while being able to perform competitively under normal conditions.\n\nS3: Theoretically, the proposed framework (RORL) enjoys a tighter suboptimality bound that PBRL.  \n\n### Weakness\n\nC1: My main complain towards the paper is that while the authors specifically tackle the tradeoff of conservatism and robustness in offline RL no clear metric has been defined to quantify the robustness of an approach. Hence the reader is forced to trust their eyes over the evaluation curves to judge the robustness of the approach. It may be insightful to invest some thoughts on quantifying the robustness by for example measuring the area under the curve on the performance under attack curve where the scale is normalized over the overall variation and dimensionality of the given dataset. \n\nC2: While this might come off as a knee jerk comment, It would be interesting to see the same set of experiments on more challenging benchmark such as ant-maze, especially as the proposed approach is claiming to better generalization ability and improve the overall robustness of the policy. Will the generalization ability result in significant improvements in a more challenging domain ? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "I5y2SE5DXDL", "reviewer": "Reviewer_frMM", "summary": "The authors handled my major concerns on approximation and experiments by providing additional responses and adding more experiments. I'd like to improve my score as boarderline accept.\n\n===\n\nTraining robust RL agent from offline datasets is an important yet challenging problem. This paper proposes RORL: offline RL algorithms with conservative smoothing. The main idea is to add smooth constraints (forcing agent to generate outputs on (adversarially) perturbed inputs) to offline RL algorithm. Here, to avoid overestimation issues, the authors also utilize uncertainties from Q-function as a penalty term. The proposed method not only achieved strong results on mujoco tasks from D4RL datasets but also showed that learned agents are more robust to perturbation.", "questions": "Q1. I like the examples for motivating the paper in Section 3 but have the following minor questions:\n\n* To obtain a perturbed state, the authors utilize action diff. However, generating perturbed state with respect to Q-function would be more natural choice for this example. Because the authors considered the perturbation with respect to Q-function in main algorithm and experiments, I wonder if there is a special reason to generate perturbed state using action diff here.\n\n* The author mentioned that they choose one of the reduced dimensions (by PCA) in Figure 3. It would be nice if the authors can clarify how to choose this dimension (for reproducibility).\n\nQ2. I have the following questions about the method:\n\n* What is the definition of perturbation set? \n\n* According to Algorithm 1, the authors mentioned that $\\hat s$ is sampled from the perturbation set but description in line 133 sounds like they found the worst case state which maximizes eq (2). This part is bit confusing. \n\n* Finding worst case state, which maximizes eq (2), is computationally expensive. Could the author clarify this part? \n", "limitations": "As pointed out in Section 8, the overhead induced by the proposed method can slow down the training. Even though this can be handled later, it would be nice if the authors also can clarify the training overhead from the proposed method (e.g. comparing training time of RORL and other offline RL algorithms). ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength\n\n* Motivation is clear and the proposed method sounds reasonable. \n\n# Weakness\n\n* Lack of evaluation on challenging domains. Even though mujoco tasks are standard benchmark in offline RL, it would be nice if the authors can evaluate on more challenging tasks such as AntMaze or Atari. Also, it would be nice if the author can consider the combination with more state-of-the-art offline RL algorithm [1].\n\n# Overall\n\n* I think this paper studies an important research question and proposes a reasonable solution. Also, the authors showed the gains from the proposed method very clearly on standard offline RL benchmarks. However, at the same time, there are several concerns (i.e., more evaluation on challenging tasks, combination with more stated-of-the-art offline RL algorithms and so on) about the draft. Because of that, I'd like to suggest \"weak reject\" but I'm also willing to change my score based on other reviews and author responses.\n\n[1] Kostrikov, I., Nair, A. and Levine, S., 2021. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CBkz6jsf8vQ", "reviewer": "Reviewer_jjFr", "summary": "This paper proposes a conservative smoothing technique by adding perturbation to the states to improve the robustness of learned policy on offline RL. Theoretically, they claim their work enjoys a tighter suboptimality bound in linear MDPs.", "questions": "More ablation study is necessary, especially for clarifying the contribution of the ensemble technique.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\nPrior offline RL methods mainly focus on the OOD actions instead of states. This paper targets the robustness issue for Q value via perturbing the states. This paper is well-written and easy to follow.\n\n**Weaknesses:**\nThe algorithm employs the ensemble Q functions. It is well known that the ensemble technique can bring robustness to learning. However, it is unclear whether or not the ensemble technique brings the main contribution to the robustness. To this end, comparing single agent methods, such as CQL and BC, is unfair. Moreover, on the results of the experiments, the proposed RORL is also very close to the ensemble baseline EDAC.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5n2xUZ87Kk2", "reviewer": "Reviewer_yLKL", "summary": "The paper investigates the problem of training robust RL agents with offline datasets. The paper claims that regularizing policy and value networks to have similar values against adversarial perturbations and applying this technique to PBRL can achieve state-of-the-art performance on the D4RL benchmarks in both standard and adversarial settings. The paper also provides a theoretical analysis of the sub-optimality gap of the proposed algorithm in linear MDPs.", "questions": "**OOD penalization**\n\nAs mentioned in line 147, the difference between the proposed method and PBRL in OOD penalization is that the proposed method penalizes both OOD states and actions. However, assuming that $\\hat{a}$ is close to $a$, the proposed method has a similar effect as PBRL since the Q-functions are trained to have similar values \bfor $s$ and $\\hat{s}$. Could you please conduct an ablation study that only penalizes OOD actions like PBRL?\n\n**Expert datasets**\n\nIs there any reason \bto exclude *-expert-v2 or *-medium-expert-v2 datasets from the experiments?\n\n**Comparison with S4RL**\n\nI think the paper is closely related to S4RL [1], which utilizes adversarial examples for smoothness. It would be better to provide the results of CQL + S4RL or PBRL + S4RL.\n\n[1] Sinha, et al., S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning in Robotics, 2021.", "limitations": "I think the proposed method is much slower than other offline RL algorithms since it has to solve mini-max problems. Please report the wall-clock time of the proposed method in the main text.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper thoroughly analyses the proposed algorithm both empirically and theoretically. \n- The proposed method achieves state-of-the-art performance in the standard offline RL benchmark, while the performance improvement is quite marginal.\n\nWeaknesses:\n- The paper does not provide experimental results on expert or near-expert datasets.\n- The proposed algorithm requires computing an adversarial example $\\hat{s}$, which is computationally expensive.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept