Robust Reinforcement Learning using Offline Data
Authors: Kishan Panaganti
Zaiyan Xu
Dileep Kalathil
Mohammad Ghavamzadeh
Venue: NeurIPS 2022 Accept (2022)
Keywords: Robust Reinforcement Learning
Offline Reinforcement Learning

Abstract: The  goal of robust reinforcement learning (RL)  is to learn a policy that is robust against the uncertainty in model parameters. Parameter uncertainty commonly occurs in many real-world RL applications due to  simulator modeling errors,  changes in the real-world system dynamics over time, and  adversarial disturbances. Robust RL is typically formulated as a max-min problem, where the objective is to learn the policy that maximizes the value  against the worst possible models that lie in an uncertainty set. In this work, we propose a  robust RL algorithm called Robust Fitted Q-Iteration (RFQI), which uses only an offline dataset to learn the optimal robust policy.  Robust RL with offline data is significantly more challenging than its non-robust counterpart because of the minimization over all models present in the robust Bellman operator. This poses challenges in offline data collection,  optimization over the models, and unbiased estimation. In this work, we propose a systematic approach to overcome these challenges, resulting in our RFQI algorithm. We prove that RFQI learns a near-optimal robust policy under standard assumptions and demonstrate its superior performance on standard benchmark problems.

Robust Reinforcement Learning using Offline Data
Kishan Panaganti', Zaiyan Xu', Dileep Kalathil', Mohammad Ghavamzadeh2
'Texas A&M University, Gooole Research.
Emails: (kpb, zxu43, dileep. kalathil)@tamu edu, ghavamza@google.comm com
Abstract
The goal of robust reinforcement learning (RL) is to learn policy that is robust against the uncertainty in model parameters. Parameter uncertainty commonly occurs in many real-world RL applications due to simulator modeling errors, changes in the real-world system dynamics over time, and adversarial disturbances. Robust RL is typically formulated as a max-min problem, where the objective is to learn the policy that maximizes the value against the worst possible models that lie in an uncertainty set. In this work, we propose a robust RL algorithm called Robust Fitted Q-Iteration (RFQI), which uses only an offline dataset to learn the optimal robust policy. Robust RL with offline data is significantly more challenging than its non-robust counterpart because of minimization over all models present in the robust Bellman operator. This poses challenges in offline data collection, optimization over the models, and unbiased estimation. this work, we propose a systematic approach overcome these challenges, resulting in RFQI algorithm. We prove that RFQI learns a near-optimal robust policy under standard assumptions and demonstrate its superior performance on standard benchmark problems.
Introduction
Reinforcement learning (RL) algorithms often require a large number of data samples to learn control policy. As result, training them directly the real-world systems is expensive and potentially dangerous. This problem is typically overcome by training them on simulator (online RL. using a pre-collected offline dataset (offline RL.. The offline dataset is usually collected either from a sophisticated simulator the real-world system or from the historical measurements. The trained RL policy is then deployed assuming that the training environment, the simulator the offline data, faithfully represents the model of the real-worid system. This assumption is often incorrect due multiple factors such the approximation errors incurred while modeling, changes in the real-world parameters over time and possible adversarial disturbances in the real-world. For example, the standard simulator settings of the sensor noise, action delay, friction, and mass of mobile robot can be different from that of the actual real-world robot, in addition to changes in the terrain, weather conditions, lighting. and obstacle densities of the testing environment. Unfortunately, the current RL control policies can fail dramatically when faced with even mild changes in the training and testing environments (Sünderhauf al., 2018; Tobin et al., 2017; Peng et al., 2018).
The goal in robust RL is to learn policy that is robust against the model parameter mismatches between the training and testing environments. The robust planning problem is formalized using the framework of Robust Markov Decision Process (RMDP) (Iyengar, 2005; Nilim and El Ghaoui, 2005). Unlike the standard MDP which considers single model (transition probability function), the RMDP formulation considers a set of models which is called the uncertainry set. The goal is to find an optimal robust policy that performs the best under the worst possible model in this uncertainty set. The minimization over the uncertainty set makes the robust MDP and robust RL problems significantly more challenging than their non-robust counterparts.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this work, we study the problem of developing a robust RL algorithm with provably optimal performance for an RMDP with arbitrarily large state spaces, using only offline data with function approximation. Before stating the contributions of our work, we provide a brief overview of the results in offline and robust RI. that are directly related to ours. We leave a more thorough discussion on related works to Appendix D.
Offfine RL: Offline RL considers the problem of learning the optimal policy only using a pre-collected (offline) dataset. Offline RL problem has been addressed extensively in the literature (Antos et al., 2008; Bertsekas, 201 1; Lange et al., 2012; Chen and Jiang, 2019; Xie and Jiang. 2020; Levine et al., 2020; Xie et al.. 2021). Many recent works develop deep RL algorithms and heuristics for the offline RL problem, focusing on the algorithmic and empirical aspects (Fujimoto et al., 2019; Kumar et al., 2019, 2020; Yu et al., 2020; Zhang and Jiang, 2021). A number of theoretical work focus on analyzing the variations of Fitted Q-Iteration (FQI) algorithm (Gordon, 1995; Ernst et al., 2005), by identifying the necessary and sufficient conditions for the learned policy to be approximately optimal and char- acterizing the performance terms of sample complexity (Munos and Szepesvári, 2008; Farahmand et al., 2010; Lazaric et al., 2012; Chen and Jiang, 2019; Liu et al., 2020; Xie et al., 2021). All these works assume that the offline data is generated according to a single model and the goal is to find the optimal policy for the MDP with the same model. In particular, none of these works consider the offline robust RL problem where the offline data is generated according to a (training) model which can be different from the one in testing. and the goal is to learn a policy that is robust w.r.t. uncertainty set.
The RMDP framework was first introduced in Iyengar (2005); Nilim and El Ghaoui (2005). The RMDP problem has been analyzed extensively in the literature (Xu and Mannor, 2010; Wiesemann et al., 2013; Yu and Xu, 2015; Mannor et al., 2016; Russel and Petrik, 2019) providing computationally efficient algorithms, but these works are limited to the planning problem. Robust RL algorithms with provable guarantees have also been proposed (Lim et al., 2013; Tamar et al., 2014; Roy et al., 2017; Panaganti and Kalathil, 2021; Wang and Zou, 2021), but they are limited to tabular or linear function approximation settings and only provide asymptotic convergence guarantees. Robust RL problem has also been addressed using deep RL methods (Pinto et al., 2017: Derman et al., 2018, 2020; Mankowitz et al., 2020; Zhang et al., 2020a). However, these works do not provide any theoretical guarantees the performance of the learned policies.
Robust RL:
The works that are closest to ours are by Zhou et al. (2021); Yang et al. (2021); Panaganti and Kalathil (2022) that address the robust RL. problem in a tabular setting under the generative model assumption. Due to the generative model assumption, the offline data has the same uniform number of samples corresponding to each and every state-action pair, and tabular setting allows the estimation of the uncertainty set followed by solving the planning problem. Our work is significantly different from these in the following way: (i) we consider a robust RL problem with arbitrary large state space, instead of the small tabular setting, (ii) we consider a true offline RL setting where the state-action pairs are sampled according an arbitrary distribution, instead of using the generative model assumption, (iii) we focus on a function approximation approach where the goal is to directly learn optimal robust value/policy using function approximation techniques, instead of solving the tabular planning problem with the estimated model. To the best of our knowledge, this is the first work that addresses the offline robust RL problem with arbitrary large state space using function approximation, with provable guarantees on the performance of the learned policy.
Offline Robust RL: Challenges and Our Contributions: Offline robust RL is significantly more challenging than its non-robust counterpart mainly because of the following key difficulties. (i) Data generation: The optimal robust policy is computed by taking the infimum over all models in the uncertainty set P. However, generating data according to all models in P is clearly infeasible. It may only be possible to get the data from a nominal (training) model Pe, How do we use the data from a nominal model to account for the behavior of all the models in the uncertainty set P?
(ii) Optimization over the uncertainty set P: The robust Bellman operator (defined in (3)) involves minimization over P, which is a significant computational challenge. Moreover, the uncertainty set P itself is unknown in the RL setting. How do we solve the optimization over ?P?
(iii) Function approximation: Approximation of the robust Bellman update requires a modified target function which also depends on the approximate solution of the optimization over the uncertainty set. How do we perform the offline RL update accounting for both approximations?
As the key technical contributions of this work, we first derive dual reformulation of the robust Bellman operator which replaces the expectation w.r.t all models in the uncertainty set P with an ex-
2

---

pectation only w.r.t. the nominal (training) model P This enables using the offline data generated by Po for learning. without relying on high variance importance sampling techniques to account for all models in P. Following the same reformulation, we then show that the optimization problem over P can be further reformulated as functional optimization. We solve this functional optimization problem using empirical risk minimization and obtain performance guarantees using the Rademacher complex- ity based bounds. We then use the approximate solution obtained from the empirical risk minimization to generate modified target samples that are then used approximate robust Bellman update through a generalized least squares approach with provably bounded errors. Performing these operations iteratively results in our proposed Robust Fitted Q-Iteration (RFQI) algorithm, for which prove that its learned policy achieves non non-asymptotic and approximately optimal performance guarantees.
Notations: For set X, we denote its cardinality as | The set of probability distribution over X' is denoted as (X and its power set sigma algebra as (X)). For any E RR we denote max (1r, 0} as (x)+- For any function f S x A -> IR, state-action distribution E ASS x A), and real number p 2 1, the w-weighted p-norm of f is defined as |l = E, (f a) 1///
2 Preliminaries
A Markov Decision Process (MDP) is tuple
where S is the state space, A is the action space, r : S x A RR is the reward function, 7 E (0, 1) is the discount factor, and do E (SS) is the initial state distribution. The transition probability function Ps.a (s') is the probability of transitioning to state s' when action a is taken at state 8. In the literature, is also called the model of the MDP. We consider setting where [S] and |A| are finite but be arbitrarily large. We will also assume that r(s, a) E [0, I. for all (s, a) E S x A, without loss of generality. A policy S + A(A) is conditional distribution over actions given a state. The value function Va, and the state-action value function Qn.P of a policy π for an MDP with model P are defined as
Vs,r(s) = E.,
En'r(s at) | so = s], Qr,((s, = E,, | rrrss at) so = =,00 a]. t=
1=0
where the expectation is over the randomness induced by the policy т and model P. The optimal value function Vp and the optimal policy xp of an MDP with the model P are defined as V/ = max, Va and u arg max, Va.r. The optimal state-action value function is given by Qpp max, Qu, The optimal policy can be obtained as (s))) = arg max, (p a). The discounted state-action occupancy of a policy for an MDP with model P. denoted dr p E A(S x A), is defined as ds, a) = (1 - 7), (2 7 (s. = 8, Og = a)}-
7'1(s
Robust Markov Decision Process (RMDP): Unlike the standard MDP which considers a single model (transition probability function), the RMDP formulation considers a set of models. We refer to this set as the uncertainty set and denote it as P. We consider P that satisfies the standard (s, a)- rectangularity condition (Lyengar, 2005). We note that a similar uncertainty set can be considered for the reward function at the expense of additional notations. However, since the analysis will be similar and the sample complexity guarantee will be identical up to a constant factor, without loss of generality, assume that the reward function is known and deterministic.
Arr, P, 7, do). where the uncertainty set P is typically defined as We specify an RMDP as M = AS,
= ((e.)ESxA Ps.a, where Ps,a (Ps.a E A(S) : D(Ps.as Pa,a) S p):
(1)
po = (Ps a (s, a) E Sx.A) is the nominal model, ((,,)) -) is distance metric between two probability distributions, and p > 0 is the radius of the uncertainty set that indicates the level of robustness. The nominal model p can be thought the model of the training environment. It is either the model trained, or in our setting, it is the model
of the simulator on which the (online) RL algorithm according to which the offline data is generated. The uncertainty set P (1) is the set of all valid transition probability functions (valid testing models) in the neighborhood of the nominal model po which by definition satisfies (s, a) -retannuularity condition (lyengar, 2005), where the neighborhood is defined using the distance metric ((:,) and radius p. In this work, we consider the Total Variation (TV) uncertainty sel defined using the TV distance, i.e., D(Pa,as Pa (1/2)) Px.a Paall..
The RMDP problem is to find the optimal robust policy which maximizes the value against the worst possible model in the uncertainty set P. The robust value function V corresponding to a policy
3

---

and the optimal robust value function V are defined as (lyengar, 2005; Nilim and El Ghaoui, 2005)
V inf V= PEP
V*= sup inf V=,P- PEP
The optimal robust policy n² is such that the robust value function corresponding to it matches the optimal robust value function, i.e., V*" = =V. It is known that there exists a deterministic optimal policy (Iyengar, 2005) for the RMDP. The robust Bellman operator is defined (lyengar, 2005)
(TQ) = r(s, a) +
inf Eg'P,, [max Q(5'.)))
Pr.aP...a P.a.
It is known that T is a contraction mapping in the infinity norm and hence it has unique fixed point Q' with (s) = max, (* a) and x =s)) = arg max, Q" (s, a) (lyengar, 2005). The Robust Q-Iteration (RQI) can now be defined using the robust Bellman operator as Qx+1 TQk- Since T is a contraction, it follows that Qs + Q'. So, RQI can be used to compute (solving the planning problem) Q" and x² in the tabular setting with a known P. Due to the optimization over the uncertainty set P... for each (s, a) pair, solving the planning problem in RMDP using RQI is much more computationally intensive than solving
MDP using Q-Iteration.
Offline RL: Offline RL considers the problem of learning the optimal policy of an MDP when the algorithm does not have direct access to the environment and cannot generate data samples in an online manner. For learning the optimal policy Tp of an MDP with model P, the algorithm will only have access to an offline dataset Dp- = {(s,, Gg Ti, } where (8,, 0z) ~ Jt, l E ASS x A) is some distribution, and s/ Ps..a.- Fitted Q-Iteration (FQI) is popular offlin RL approach which is amenable to theoretical analysis while achieving impressive empirical performance. In addition to the dataset Dp, FQI uses a function class F { : S x A + [0, - }]]3 to approximate Q7- The typical FQI update is given by fk+1 = arg | Σx= (r(ss, a,) + y max; fi((s,, b) fs, a;))?, which aims to approximate the non-robust Bellman update using offline data with function approximation. Under suitable assumptions, is possible obtain provable performance guarantees for FQI (Szepesvári and Munos, 2005; Chen and Jiang, 2019; Liu et al., 2020).
3 Offline Robust Reinforcement Learning
The goal of an offline robust RL algorithm is to learn the optimal robust policy x² using a pre-collected offline dataset D. The data is typically generated according to a nominal (training) model P i.e., D {ss, 0g Pi, } 1= where (s;, a,) ~ μ.u. E A(S A) is some data generating distribution, and s ~ Po s...." The uncertainty set P is defined around this nominal model p as given in (1) w.r.t. the total variation distance metric. We emphasize that the learning algorithm does not know the nominal model p as it has only access to D, and hence it also does not know P. Moreover, the learning algorithm does not have data generated according to any other models in and has to rely only on D to account for the behavior w.r.t. all models in P.
Learning policies for RL problems with large state-action spaces computationally intractable. RL algorithms typically overcome this issue by using function approximation. In this paper, we consider two function classes J (J: S x A - [0, 1/(1 -)]} and g = {g : S x A + 2/((1((( - -]]]]- We use F to approximate Q' and G to approximate the dual variable functions which we will introduce in the next section. For simplicity, we will first assume that these function classes are finite but exponentially large, and we will use the standard log-cardinality to characterize the sample complexity results, as given in Theorem 1. We note that, at the cost of additional notations and analysis, infinite function classes can also be considered where the log-cardinalities are replaced by the appropriate notions of covering number.
Similar to the non-robust offline RL, we make the following standard assumptions about the data generating distribution μ and the representation power of F.
Assumption 1 (Concentratabillity) There exists finite constant C 0 such that for any v e (dx, any policy T C ASS x A), we have ||//||/o < VC.
Assumption states that the ratio of the distribution v and the data generating distribution je, v(s, a)/u(s, a), is uniformly bounded. This assumption is widely used in the offline literature (Munos, 2003; Agarwal et al., 2019; Chen and Jiang, 2019; Wang et al., 2021; Xie et al., 2021) in many different forms. We borrow this assumption from Chen and Jiang (2019), where they used it for

---

non-robust offline RL. In particular, we note that the distribution 1/ is in the collection of discounted state-action occupancies on model pe alone for the robust RL. Assumption 2 (Approximate completeness), Let μ E A(S A) be the data distribution. Then, PEF supser PEF II." T S Ec.
Assumption 2 states that the function class F is approximately closed under the robust Bellman operator T. This assumption has also been widely used in the offline RL literature (Agarwal et al., 2019: Chen and Jiang, 2019; Wang al., 2021; Xie et al., 2021).
One of the most important properties that the function class F should have is that there must exist a function I' F which well-approximates Q" This assumption is typically called approximate realiz- ability in the offline RL literature. This typically formalized by assuming inl IIʃ - T fll S & (Chen and Jiang, 2019). It is known that the approximate completeness assumption and the concen- tratability assumption imply the realizability assumption (Chen and Jiang, 2019; Xie et al., 2021).
4 Robust Fitted Q-Iteration: Algorithm and Main Results
In this section, we give a step-by-step approach to overcome the challenges of the offline robust RL outlined in Section 1. We then combine these intermediate steps to obtain our proposed RFQI algorithm. We then present our main result about the performance guarantee of the RFQI algorithm, followed by brief description about the proof approach.
4.1 Dual Reformulation of Robust Bellman Operator
One key challenge in directly using the standard definition of the optimal robust value function given in (2) or of the robust Bellman operator given in (3) for developing and analyzing robust RL algorithms is that both involve computing an expectation w.r.t. each model P. Given that the data is generated only according to the nominal model pa, estimating these expectation values is really challenging. We show that we can overcome this difficulty through the dual reformulation of the robust Bellman operator, as given below.
Proposition 1. Let M be an RMDP with the uncertainty set P specified by (1) using the total variation distance D(Ps.a s,a1 po.0)= = (1/2)| -o Paaalla: Then, for any Q : S A + [0, 1/(1 7)],
the robust Bellman operator T given in (3) can be equivalently written as
inf (TQ) a) = r(s. a) neto, ((-T)
(Eo-re, [ V(x'))+) - + p(n - inf V )))+ (4)
where (s)) = m Q(, a). Moreover, the inner optimization problem in (4) is convex in
This result mainly relies on Shapiro (2017, Section 3.2) and Duchi and Namkoong (2018, Proposition
1). Note that in (4), the expectation is now only w.r.t. the nominal model p which opens up the possibility of using empirical estimates obtained from the data generated according to po, This avoids the need to use importance sampling based techniques to account for all models in P, which often have high variance. and thus, are not desirable.
While (4) provides a form that is amenable to estimation using offline data, it involves finding inf ((s"). Though this computation is straightforward in tabular setting, it is infeasible in function approximation setting. In order to overcome this issue, we make the following assumption.
Assumption (Pail-state). The RMDP M has a 'fail-state' 8f, such that r(sy,a) (z)) 1, Va E A, vP E P.
and
We note that this is not a very restrictive assumption because such a 'fail-ttate' is quite natural in most simulated or real-world systems. For example, a state where robot collapses and is not able to get up, either in a simulation environment like MuloCo or in real-world setting, is such a fail state.
Assumption 3 immediately implies that Vs,)(sy) O, VP E P, and hence V (sy) = 0 and Q' a) = 0, Va E A. It is also straightforward to see that Q841(88,0)
0, Va E A. where Q s are the RQI iterates given by the robust Bellman update Q++1 = TQ² with the initialization Qo = O. By the contraction property of T, we have Qi - Q* So, under Assumption 3, without loss of generality, we can always keep Qu(sy, a) O, Va A and for all A in ROI (and later in RFQI).
5

---

So. in the light of the above description, for the rest of the paper we will use the robust Bellman
operator T by setting inf V(s") = O. In particular, for any function f f:S x A + [0. 1/ - 7)]
with [8 a) = 0, the robust Bellman operator 7 is now given by inf
(Exr [(1 - max a' (T (T a) = r(8, a) ne(0 (ot--))
[( ,)) (1 - p)v). (5)
4.2 Approximately Solving the Dual Optimization using Empirical Risk Minimization
Another key challenge in directly using the standard definition of the optimal robust value function given in (2) of the robust Bellman operator given in (3) for developing and analyzing robust RL algorithms is that both involve an optimization over P. The dual reformulation given in (5) partially overcomes this challenge also, the optimization over P is now replaced by convex optimization over a scalar 7) E 2/ [/[[]] However, this still requires solving an optimization for each (s, a) E S A. which clearly infeasible even for moderately sized state-action spaces, not to mention the function approximation setting. Our key idea to overcome this difficulty is to reformulate this as a functional optimization problem instead of solving it as multiple scalar optimization problems. This functional optimization method will make it amenable to approximately solving the dual problem using an empirical risk minimization approach with offline data.
Consider the probability (measure) space (S x A, E(S x A). and let (S x A, E(S x A)), 1)) be the set of all absolutely integrable functions defined on this space. In other words, I.¹ is the set of all functions g S x A + C C I. such that || I.p. is finite. We set C = p/ - 7)], anticipating the solution of the dual optimization problem (5). We also note is the data generating distribution which is a o-finite measure.
For any given function f : x - [0, 1/(1 - 7]], we define the loss function [=)a as Tg f) = Es [Ep' [[g(s, a) max a(s), - (1 - ())(pp)(ss) a)), Vgg E T,t. (6)
In the following lemma, we show that the scalar optimization over 17 for each (s,a) a) pair in (5) can be replaced by a single functional optimization w.r.t. the loss function Ldual.
Lemma Let Laual be the loss function defined in (6). Then, for any function f S x 1/(1 7)}, we have
inf Lauaig;: f) = - Byara gEL
inf Lnjo (al-n))
(Bgwngg [(1 max f(s', f(s', a')) - - a'
(7)
Note that the RHS of (7) has minimization over v) for each (x, a) pair and minimization is inside the expectation Es,a-pl-] However, the LHS of (7) has a single functional minimization over y E L and this minimization is outside the expectation. For interchanging the expectation and minimization, and for moving from point-wise optimization to functional optimization, we use the result from Rockafellar and Wets (2009, Theorem 14,60), along with the fact that L is decomposable space. We also note that this result has been used in many recent works distributionally robust optimization (Shapiro, 2017; Duchi and Namkoong, 2018) (see Appendix A for more details).
We can now define the empirical loss function Lranal corresponding to the true loss Lduat as
N
Lanal (gr 1)
1 N
E (g(os,, - max l's(,, - -11- (1-p)g(s;,a))
a
(8)
d
Now, for any given f. we can find an approximately optimal dual function through the empirical risk minimization approach as gnf Lawat(g: f).
As we mentioned in Section 3, our offline robust RL algorithm is given an input function class ( : S x A [0, 2/ (p(1 }})}} to approximate the dual variable functions. So, in the empirical risk minimization, instead of taking the infimum over all the functions in I,=, we can only take the infimum over all the functions in G. For this be meaningful, should have sufficient representation power. In particular, the result in Lemma should hold approximately even if we replace the infimum over L with infimum over G. One can see that this is similar the realizability requirement for the function class F as described in Section 3. We formalize the representation power of & in the following assumption.
In the following, will simply denote L'(S A, LIS x A). as L for conciseness.

---

Assumption 4 (Approximate dual realizability). For all f E F, there exists uniform constant dduai such that inf geg Laual (g: f) inf Lana 9: f) ddual
Using the above assumption, for any given f E F, can find an approximately optimal dual function ōn E G through the empirical risk minimization approach as Ūy arg min Lduat f) In order to characterize the performance of this approach, consider the operator Tg for any g E G as f(s,, (odd.a))) -'))+1 p)g(s,
(s, r((s,) Pr, ((g(s, a) a'
(9)
for all f F and (s, a) E A. We will show in Lemma in Appendix C that the error sur [T - Tă f 1 is ((log(()//////]])) with probability at least 1 8.
4.3 Robust Fitted Q-iteration
The intuitive idea behind robust fitted Q-iteration (RFQI) algorithm is to approximate the exact RQI update step Qk-1 1 = TQx with function approximation using offline data. The exact RQI step requires updating each (8, aa-pair separately, which scalable to large state-action spaces, So, this is replaced by the function approximation as Qk+1 arg minjer [TQ; fll It is still infeasible to perform this update it requires to exactly compute the expectation (w.r.t po and μ) and to solve the dual problem accurately. We overcome these issues by replacing both these exact computations with empirical estimates using the offline data. We note that this intuitive idea is similar to that of the FQI algorithm in the non-robust case. However, RFQI has unique challenges due to the nature of the robust Bellman operator T and the presence of the dual optimization problem within T.
Given a dataset D, we also follow the standard non-robust offline RL. choice of least-squares residual minimization (Chen and Jiang, 2019; Xie et al., 2021; Wang et al., 2021). Define the empirical loss of f given f' (which represents the Q-function from the last iteration) and dual variable function g as
1 N Σ
r(8;, a:) + 7( (g(ss, a;) max ('(8,, a')) + (1 (lg(ss, f(s, 0:)
)²
(10)
The correct dual variable function to be used in (10) is the optimal dual variable 97. arg arg Lami(g: f') corresponding to the last iterate f", which we will approximate it by 9r = arg min Ldual(g; f'). The RFQI update is then obtained as arg minjer Lf ff )r.).
Summarizing the individual steps described above, we formally give our RFQI algorithm below.
Algorithm 1 Robust Fitted Q-Iteration (RFQI) Algorithm
1: Input: Offline dataset D = (s,, Q, Fi, s function classes F and G.
2: Initialize: Qu E
3: for k == ..... K 1 do
4: Dual variable function optimization: Compute the dual variable function corresponding to
Qs through empirical risk minimization as 9k = To. = arg min Lauul(g; Qx) (see (8)).
5: Robust Q-update: Compute the next iterate Qk+1 through least-squares regression as
Qu+1 arg minger LRFQI(Q; gk.gk) (see (10)).
6: end for
7: Output: मx arg max, Qs(s,a)
Now state our main theoretical result on the performance of the RFQI algorithm.
Theorem Let Assumptions 1-4 hold. Let æ be the ouppu of the RFQI algorithu after K iterations.
Denote J = Exdd, x (s)] where dy is initial state distribution. Then, for any E (0, 1), with probability least 1 28, we have
2
VOlybe, + (edmal
16
18C log (2 (2 (
n JK <
+
+
[1 7
(1-7)2
p(1 - 33
N
7

---

Remark 1. Theorem 1 states that the RFQI algorithm can achieve approximate optimality. To see this, note that with K 2 O( [og(1/7] log( )). and neglecting the second term corresponding to (inevitable) approximation errors 6c and ddall. we get J = S e/(1 7) with probability greater than 1 26 for any e, E (0, 1). as long as the number of samples N 2 O( (o=)²(1->>4 log 5 So, the above theorem can also be interpreted as a sample complexity result.
(
Remark 2. The known sample complexity of Bobust-RL in the tabular setting is O ISP
(pe)=(1-7) 734 (Yang et al., 2021; Panaganti and Kalathil, 2022). Considering (1og((FF]()) to be ((S))A)), we can recover the same bound as the tabular setting (we save |S] due to the use of Bernstein inequality). Remark Under similar Bellman completeness and concentratability assumptions, RFQI sample complexity is comparable to that of a non-robust offline RI. algorithm, i.e., Ol (( log _ (Chen and Jiang, 2019). As consequence of robustness, we have pー² and log([G]) factors in our bound.
4.4 Proof Sketch
Here we briefly explain the key ideas used in the analysis of RFQI for obtaining the optimality gap bound in Theorem 1. The complete proof is provided in Appendix C.
Step 1: To bound J -J we connect it to the error I|Q -Q I12.0 for any state-action distribution
While the similar step follows almost immediately using the well-known performance lemma in the analysis of non- non-robust FQI, such a result is not known in the robust RL setting. So, we derive the basic
inequalities to get a recursive form and to obtain the bound / - J 2 -K (see (22) and the steps before in Appendix C).
Step 2: To bound |!QT Qx 11. for any state-action distribution such that ||//|| S VC, decompose to get a recursion, with approximation terms based on the least-squares regression and empirical risk minimization. Recall that 9y is the dual variable function from the algorithm for state- action value function f E Denote Jú as the least squares solution from the algorithm for the state- action value function f E F and dual variable function g E G, i.e., fg = arg mings LurQu (Q; f. J.9). By recursive use of the obtained inequality (23) (see Appendix C) and using uniform bound, we get
K
VC
sup [ T3 fll., + VC
-KKIIP S
+ 1 SEF
sup sup
|Tp - l2
fEF geG
Step 3: We recognize that sup IT f is an empirical risk minimization error term. Using Rademacher complexity based bounds, we show in Lemma 6 that this error is (llg((FF/66)/VV) with high probability.
Step 4: Similarly, we also recognize that sup supges |Ty - 5gll2, is a least-squares regression error term. We also show that this error is (log([F]]((/)) vN) with high probability. We adapt the generalized least squares regression result to accommodate the modified target functions resulting from the robust Bellman operator to obtain this bound (see Lemma 7). The proof is complete after combining steps 1-4 above.
5 Experiments
"force, mag" perturbation
Action parturtattion
"leg jaapjjoontttiiiii stiffines" perturbation
323
X7Q
123
3000
140
FFQ
43-
Percentage change from nominal value
Prob. of picking random action
1eg_joint stifhese values Idefault-0.00
Figure 1: CartPole
Figure 2: CartPole
Figure 3: Hopper
Here, we demonstrate the robust performance of our RFQI algorithm by evaluating on Cartpole and Hopper environments in OpenAI Gym (Brockman et al., 2016). In all the figures shown, the quantity
8

---

in the vertical axis is averaged over 20 different seeded runs depicted by the thick line and the band around it is the =0.5 standard deviation. more detailed description of the experiments, and results on additional experiments, are deferred to Appendix F. We provide our code in github webpage https: www../////aaaa..aeee IRFQI containing instructions to reproduce all results in this paper.
For the Cartpole, we compare RFQI algorithm against the non-robust RL algorithms FQI and DQN, and the soft-robust RL algorithm proposed in Derman et al. (2018). We test the robustness of the algorithms by changing the parameter force_ (to model external force disturbance), and also by introducing action perturbations (to model actuator noise). Fig. 1 and Fig. shows superior robust per- formance of RFQI compared to the non-robust FQI and DQN. The RFQI performance is similar to that of soft-robust DQN. We note that soft-robust RL algorithm (here soft-robust DQN) is an online deep RL algorithm (and not an offline RL algorithm) and has no provable performance guarantee. More- over, soft-robust RI. algorithm requires generating online data according number of models in the uncertainty set, whereas RFQI only requires offline data according to a single nominal training model.
For the Hopper, we compare RFQI algorithm against the non-robust RL algorithms FQI and TD3 (Fujimoto et al., 2018) 8), and the soft-robust RL (here soft-robust DDPG) algorithm proposed in Derman al. (2018). We test the robustness of the algorithms by changing the parameter "rs_joonnsssffieee Fig. 3 shows the superior performance of our RFQI algorithm against the non-robust algorithms and soft-robust DDPG algorithm. The average episodic reward of RFQI remains almost the same initially, and later decays much less and gracefully when compared to the non-robust FQI and TD3.
6 Conclusion
In this work, we presented novel robust RL algorithm called Robust Fitted Q-Iteration algorithm with provably optimal performance for an RMDP with arbitrarily large state space, using only offline data with function approximation. We also demonstrated the superior performance of the proposed algorithm on standard benchmark problems.
One limitation of our present work is that, we considered only the uncertainty set defined with respect to the total variation distance. In future work, we will consider uncertainty sets defined with respect to other f-divergences such as KL-divergence and Chi-square divergence, Finding lower bound for the sample complexity and relaxing the assumptions used are also important and challenging problems.
7
Acknowledgements
This work was supported in part by the National Science Foundation (NSF) grants NSF-CAREER- EPCN-2045783 and NSF ECCS 2038963. Any opinions, findings, and conclusions or recommenda- tions expressed in this material are those of the authors and do not necessarily reflect the views of the sponsoring agencies.
References
Agarwal, A., Jiang. N., Kakade, S. M., and Sun, W. (2019). Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep. 4, 5, 19. 21
Agarwal, A., Kakade, S., and Yang, L. F. (2020). Model-based reinforcement learning with generative model is minimax optimal. In Conference on Learning Theory, pages 67-83. 23
Antos, A., Szepesvári, C., and Munos, R. (2008). Learning near-optimal policies with Bellman- residual minimization based fitted policy iteration and single sample path. Machine Learning, 71(1):89-129. 2, 23
Azar, M. G., Munos, R., and Kappen, H. J. (2013). Minimax PAC bounds on the sample complexity of reinforcement learning with generative model. Mach. Learn.. 91(3):325- 349. 23
Bertsekas, D. P. (2011). Approximate policy iteration: A survey and some new methods. Journal of Control Theory and Applications, 9(3):3 10-335 2, 23
Borkar, V. S. (2002). Q-learning for risk-sensitive control. Mathematies of operations research, 27(2):294-311. 23

---

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv preprint arXiv:1606.01540. 24, 27
Chen, J. and Jiang, N. (2019). Information-theeretiic considerations in batch reinforcement learning. In International Conference on Machine Learning, pages 1042-1051. 2, 4, 8, 23
Csiszár, I. (1967). Information-type measures of difference of probability distributions and indirect observation. studia scientiarum Mathematicarum Hungarica, 2:229-318. 16
Derman, E., Mankowitz, D., Mann, T., and Mannor, S. (2020). bayesian approach to robust
reinforcement learning. In Uncertainty in Artificial Intelligence, pages 648-658. 2
Derman, E., Mankowitz, D. J., Mann, T. A., and Mannor, S. (2018), Soft-robust actor-critic policy- gradient. In AUAI press for Association for Uncertainty in Artificial Intelligence, pages 208 218. 2, 9, 23, 26
Duchi, J. and Namkoong, H. (2018). Learning models with uniform performance via distributionally robust optimization. arXiv preprint arXiv:1810.08755 5, 16
Dullerud, G. E. and Paganini, F. (2013). A course in robust control theory: a convex approach,
volume 36. Springer Science & Business Media. 23
Ernst, D., Geurts, P., and Wehenkel, L. (2005). Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2, 23
Farahmand, A.-m., Szepesvári, C., and Munos, R. (2010). Error propagation for approximate policy and value iteration. Advances in Neural Information Processing Systems, 23. 2, 23
Fei, Y., Yang, Z., Chen, Y., and Wang. Z. (2021). Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning. In Annual Conference on Neural Information Processing Systems 2021, pages 20436-20446. 23
J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. (2020). D4rl: Datasets for deep data-driven reinforcement learning. 26, 27, 28
Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic
methods. In International Conference on Machine Learning, pages 1582-1591. 9, 27
Fujimoto, S., Meger, D., and Precup, D. (2019). Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. 2, 23, 24, 25, 27
Gordon, G. (1995). Stable function approximation in dynamic programming. Machine learning proceedings 1995, pages 261-268. 2, 23
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. 25,
Haskell, W. B., Jain, R., and Kalathil, D. (2016). Empirical dynamic programming. Mathematics of Operations Research, 41(2):402-429. 23
Huang, P., Xu, M., Fang, F., and Zhao, D. (2022). Robust reinforcement learning as a stackelberg
game via reaplivvly-rrgullarzedd adversarial training. arXiv preprint arXiv:2202.09514, 23
Iyengar, G. N. (2005). Robust dynamic programming. 30(2):257-280. 1, 3, 21, 23
Mathematics of Operations Research,
Kalathil, D., Borkar, V. S., and Jain, R. (2021). Empirical Q- Q-Value Iteration. Stochastic Systems, 11(1):1-18. 23
Kaufman, I.. and Schaefer, A. J. (2013). Robust modified policy iteration. INFORMS Journal on Computing, 25(3):396-44 23
10

---

Kingma, D. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv:1412.699 24
arXiv preprint
Kingma, D. and Welling, M. (2013), Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.6114. 24
Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. (2019). Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11784-11794. 2, 23
Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020). Conservative q-learning for offline reinforcement learning, Advances in Neural Information Processing Systems, 33:1179-1191. 2, 23
Lange, S., Gabel, T., and Riedmiller, M. (2012). Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer. 2, 23
Lazaric, A., Ghavamzadeh, M., and Munos, R. (2012). Finite-sample analysis of least-squares policy iteration. Journal of Machine Learning Research, 13:3041-3074, 2, 23
Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2, 23, 26, 27, 28
Li, G., Wei, Y., Chi, Y., Gu, Y., and Chen, Y. (2020). Breaking the sample size barrier in model-based reinforcement learning with a generative model. In Advances in Neural Information Processing Systems, volume 33, pages 12861-12872 23
Lim, S. H. and Autef, A. (2019). Kernel-based reinforcement learning in robust Markov decision processes. In International Conference on Machine Learning, pages 3973-3981. 23
Lim, S. H., Xu, H., and Mannor, S. (2013). Reinforcement learning robust Markov decision processes. In Advances in Neural Information Processing Systems, pages 701-709. 2
Liu. Y.. Swaminathan, A., Agarwal, A., and Brunskill, E. (2020). Provably good batch off-policy reinforcement learning without great exploration. In Neural Information Processing Systems. 2, 4, 23, 24, 25, 26, 27,20
Mankowitz, D. J., Levine, N., Jeong, R., Abdolmalcki, A., Springenberg. J. T., Shi. Y., Kay, J., Hester, T., Mann, T., and Riedmiller, M. (2020). Robust reinforcement learning for continuous control with model misspecification. In International Conference on Learning Representations. 2, 23
Mannor, S., Mebel, O., and Xu, H. (2016). Robust mdps with k-rectangular uncertainty. Mathematics of Operations Research, 4144)1148-1150
Moses, A. K. and Sundaresan, (2011). Further results on geometric properties of family of relative entropies. In 2011 IEEE International Symposium on Information Theory Proceedings, pages 1940-1944. 16
Munos, R. (2003). Error bounds for approximate policy iteration. ICML, volume 3, pages 560-567. 4
Munos, R. and Szepesvári, C. (2008). Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(27)8115-87 2, 23
Nilim, A. and El Ghaoui, L. (2005). Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 1, 2, 4, 23
Panaganti, K. and Kalathil, D. (2021). Robust reinforcement learning using least squares policy itera- tion with provable performance guarantees. In Proceedings of the 38th International Conference
on Machine Learning, pages 51 511-520. 2, 23
Panaganti, K. and Kalathil, D. (2022). Sample complexity of robust reinforcement learning with generative model. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pages 9582-9602. 2, 8,8, 24

---

Peng, X. B.. Andrychowicz, M., Zaremba, W., and Abbeel, P. (2018). Sim-to-real transfer of robotic control with dynamies randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1800-3810. IEEE.
Pinto, L.. Davidson, J., Sukthankar, R., and Gupta, A. (2017). Robust adversarial reinforcement
learning. International Conference on Machine Learning, pages 2817-2826. 2, 23
Prashanth, L. A. and Ghavamzadeh, M. (2016). Variance-constraanee actor-critic algorithms for discounted and average reward mdps. Mach. Leamn,, 105(3):367-477. 23
Raffin, A. (2020). RI baselines3 700. https: bttps://githut wwwpethhu............eee com/DLR-RM// sttll-aaselnnee---oeoe 25
Rockafellar, R. T. and Wets, R. J.-B. (2009). Variational analysis, volume 317. Springer Science & Business Media. 6, 15, 16
Roy, A., Xu, H., and Pokutta, S. (2017). Reinforcement learning under model mismatch. In Advances in Neural Information Processing Systems, pages 3043-3052. 23
Russel, R. H. and Petrik, M. (2019). Beyond confidence regions: Tight bayesian ambiguity sets for robust mdps. Advances in Neural Information Processing Systems." 2
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. 25
Shalev-Shwartz, S. and Ben-David, S. (2014). Understanding machine learning: From theory to algorithms. Cambridge university press. 15
Shapiro, A. (2017). Distributionally robust stochastic programming. SIAM Journal on Optimization,
27(4):2258-2275. 5, 6, 16
Sidford, A., Wang, M., Wu, X., Yang. L. F., and Ye, Y. (2018). Near-optimal time and sample complexities for solving markov decision processes with generative model. Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages
23
Singh, S. P. and Yee, R. C. (1994). An upper bound on the loss from approximate optimal-value functions. Machine Learning, 16(3):227-233 23
Sünderhauf, N., Brock, O., Scheirer, W., Hadsell, R., Fox, D., Leitner, J., Uperoft, B., Abbeel, P., Burgard, W., Milford, M., et al. (2018). The limits and potentials of deep learning for robotics. The International journal of robotics research, 37(4-5):405-421
Szepesvári, C. and Munos, R. (2005). Finite time bounds for sampling based fitted value iteration. In Proceedings of the 22nd international conference Machine learning, pages 880-887. 4
Tamar, A., Mannor, S., and Xu, H. (2014). Scaling up robust mdps using function approximation. In International Conference on Machine Learning, pages 181-189 2, 23
Tessler, C., Efroni, Y., and Mannor, S. (2019). Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages 6215 62224. 23
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RS] international conference on intelligent robots and systems (IROS), pages 23-30.
Vershynin, R. (2018). High-Dimensional Probability: An Introduction with Applications in Data Science, volume 47. Cambridge University press. 15
Wang, R., Foster, D., and Kakade, S. M. (2021). What are the statistical limits of offline {rl) with linear function approximation? In International Conference on Learning Representations. 4, 5,
Wang, Y. and Zou, S. (2021). Online robust reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems, 34. 2
Wiesemann, W., Kuhn, D., and Rustem, B. (2013). Robust Markov decision processes. Mathematics of Operations Research, 38(1):153-183, 2, 23
12

---

Xie. T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. (2021). Bellman- consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34. 2, 5, 23
Xie, T. and Jiang, N. (2020). Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence, pages 550-559. 2, 23
Xu, H. and Mannor, S. (2010). Distributionally robust Markov decision processes. In Advances in Neural Information Processing Systems, pages 2505-2513. 2, 23
Yang. W., Zhang, L., and Zhang, Z. (2021). Towards theoretical understandings of robust markoy decision processes: Sample complexity and asymptoties. arXiv preprint arXiv:2105.03863. 2, 24
Yu, P. and Xu, H. (2015). Distributionally robust counterpart in Markov decision processes. IEFF Transactions on Automatic Control, (1(9):2538 2543.
Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., and Ma. T. (2020). Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems. 2, 23
Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., and Hsieh, C.-J. (2020a). Robust deep reinforcement learning against adversarial perturbations state observations. Advances in Neural Information Processing Systems, 33:21024-21037.
Zhang, K., Hu, B., and Basar, T. (2020b). Policy optimization for H2 linear control with Hoc robustness guarantee: Implicit regularization and global convergence. Proceedings of the 2nd Annual Conference on Learning for Dynamics and Control, volume 120, pages 179-190. 23
Zhang, S. and Jiang, N. (2021). Towards hyperparameter-fiee policy selection for offline reinforce- ment learning. Advances Neural Information Processing Systems, pages 12864-12875.
23
Zhang, X., Chen, Y., Zhu, X., and Sun, W. (2022). Corruption offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 5757-5773. PML. 23
Zhang, Y., Yang, Z., and Wang. Z. (2021). Provably efficient actor-critic for risk-sensitive and robust adversarial rl: A linear-quadratic case. In International Conference on Artificial Intelligence and Statistics, pages 2764-2772. 23
Zhou, K., Doyle, J. C., Glover, K., et al. (1996). Robust and optimal control, volume 40. Prentice hall New Jersey. 23
Zhou, Z., Bai, Q., Zhou, Z., Qiu, L., Blanchet, J., and Glynn, P. (2021). Finite sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference Artificial Intelligence and Statistics, pages 3331-3339. 24
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See contributions in the Introduction.
(b) Did you describe the limitations of your work? [Yes] The discussions on the assump- tions describes the limitations.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? (Yes) See Sections 3-4.3
(b) Did you include complete proofs of all theoretical results? [Yes] We provide proof
sketch 4.4 in main paper and the complete proof in Appendix with self-contained material.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Described in the Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi-
ments multiple times)? [Ye]] Described in the main paper and the Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Mentioned the Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "tM9zg0mJGjG", "reviewer": "Reviewer_NZk6", "summary": "The paper deals with robust offline RL using function approximators. Robustness is a broad term, here it means a special robustness defined in the RMDP framework. An algorithm is outlined and numerous proofs of the properties of the algorithm are proved under strong assumptions. The algorithm is evaluated on two deterministic benchmarks (cart pole and hopper). \nRobustness to changes in system dynamics is investigated, and promising performance is achieved on the two selected benchmarks relative to the selected comparison algorithms.\n", "questions": "The proofs make several assumptions, it would be interesting to see to what extent these assumptions are met in practice. Specifically, it would be interesting to see how often it happens in the experiments that the assumptions of the proofs are fulfilled.\n\nIn Figure 1 and 2 the algorithms FQI, DQN, SR-DQN and RFQI are examined, in Figure 3 it is FQI, TD3, SRDDPG and RQI. Why were the same algorithms not compared on all benchmarks? In addition, the same colors are used in the three figures to identify the algorithms, although they are not the same algorithms. This should be explained or changed.\n\nIn Figure 3, it is noticeable that for small values of \"lag_joint_stiffness\" of 0, 5 and 10 TD3 is clearly superior to the proposed RFQI. This should be mentioned in the text. Possibly it should be mentioned as a limitation. Also, it should be explained why this does not contradict claim „we presented a novel robust RL algorithm called Robust Fitted Q-Iteration algorithm with provably optimal performance for an RMDP with arbitrarily large state space, using only offline data with function approximation“. Furthermore, claim „We also demonstrated the superior performance of the proposed algorithm on standard benchmark problems“ seems to be an exaggeration.\n\n„Parameter uncertainty commonly occurs in many real-world RL applications due to simulator modeling errors, changes in the real-world system dynamics over time, and ad\\ersarial disturbances“\\\nI am skeptical that \"Parameter uncertainty commonly occurs in many real-world RL applications due to ... adversarial disturbances\". I don't think you can write it that way without giving a reference to the source of this assessment.\n\n\nLesser importance:\\\n„Robust RL is typically formulated as a max-min problem“\\\nI would rather say this is one variant, others use percentile performance optimization.\n\n\"Learning policies for RL problems with large state-action spaces is computationally intractable.\"\\\nThis claim is not correct in this form (backgammon, go, chess, manifold applications with continuous state and action spaces).\n\nIn the bibliography there are a few wrong lower case letters, like: bayesian, mdp \n\n\n\n", "limitations": "There are, in my opinion, a few limitations that should still be addressed in the text.\n\nTo what extent the conditions for the validity of the proofs are realized in practice should be discussed as a possible limitation.\n\nFurthermore, the limitation that the benchmarks used have deterministic dynamics throughout should be mentioned.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The paper makes extensive theoretical contributions.\n* The proposed practical algorithm could be promising.\n\n**Weaknesses**\n* The paper makes exaggerated claims.\\\n\"In this work, we presented a novel robust RL algorithm called Robust Fitted Q-Iteration algorithm\nwith provably optimal performance for an RMDP with arbitrarily large state space, using only offline data with function approximation.\"\\\nOptimal? Provably optimal? The limitations and assumptions should be named. The non-optimality evident in Figure 3 should be explained.\\\n„We also demonstrated the superior performance of the proposed algorithm on standard benchmark problems.“\\\nBased on figures 1, 2 and 3, there is too little evidence, that the performance is superior. Furthermore, the statement can only be made for the examined alternative algorithms, furthermore there are only two benchmarks.\n\n* The empirical evidence of the usefulness of the method is weak.\\\nOnly two benchmarks, moreover both contain only deterministic dynamics. Due to the lack of investigations on benchmarks with stochastic dynamics, it seems questionable to me whether the experiments are at all suitable to test the claim of the proposed method.\n* The paper seems incomplete without the Appendix.\\\nAlgotithm 2. More experiments.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "q6zBUFqlwwf", "reviewer": "Reviewer_3j8B", "summary": "This paper endeavors to design a robust algorithm in the offline RL setting with function approximation. The proposed algorithm is called Robust Fitted Q-Iteration, hopefully addressing challenges in this specific setting. The detailed derivation of the proposal is given under certain assumptions, followed by a near-optimal guarantee or sample complexity result. Experiments are conducted on three typical environments.\n\n", "questions": "1.In Eq.10, I am wondering why seeking the optimal dual function $g$ is equivalent to an optimal dual variable, and what the approximation gap between $\\hat{g_{f^\\prime}}$ and $g^*_{f^\\prime}$?\n\n2.The paper [1] also considers the robust RL setting, so what is the difference between it with this paper? Relevant discussion should be added in the related work.\n\n3.Three challenges mentioned in Section 1 seems only to involve Robust RL. If additionally considering the offline setting, is there any new challenge for offline robust RL?\n\n[1] Corruption-Robust Offline Reinforcement Learning (AISTAT 2022)\n", "limitations": "1.The paper designs the robust algorithm only based on Total variation, which might be limited in terms of practice. I am not sure why the authors chose this distance rather than KL or others? Is TV the most typical choice in robust RL literature? An explanation is expected in the future.\n\n2.The empirical improvement over other baselines seems not to be significant. More experiments are needed, including on more games. Also, a detailed comparison of the computational cost is also suggested.\n\nIn summary, I may not be familiar with some technical details, but as far as I can tell, they are basically sound. In particular, the dual reformulation as well as the following assumptions serves as the key to the proposed algorithm, which is the critical contribution of this paper. Therefore, I think it is a borderline paper as the algorithmic contribution is notable, while the practicability of the proposed algorithms might be questionable and needs to be further demonstrated.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength.\n1.Related works and preliminaries are thorough and comprehensive.\n\n2.The paper is well organized and easy to follow.\n\n\n### Weakness\n1.In the design process of Robust Fitted Q-iteration, the sequentially added assumptions, e.g., Assumptions 3 and 4, lack demonstration empirically. In practice, we are not fully sure whether these assumptions can be satisfied or not, and how they impact on empirical performance.\n\n2.Emprical demonstration is insufficient as authors only compare their algorithm with a limited number of baselines, e.g., SR-DQN, on only two or three Mujoco games. Researchers tend to be concerned about whether the empirical superior performance of the proposed algorithm can be observed as well on other games.\n\n3.The proof sketch might be wordy and hard to understand.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "C3Kl-GGLxIw", "reviewer": "Reviewer_NXv3", "summary": "This paper suggests a method to solve robust MDP using offline data. A fitted Q-learning is proposed and the authors developed its convergence and sample complexity. ", "questions": "(1). See Weakness above. \n(2). Assumption 4 assumes the representative ablity of the function class, can the authors provide some example? Linear function approximation? Neural network? \n(3) Assumption 1 assumes an upper bound of $\\mu$, and hence we have $\\min_{s,a} \\mu(s,a)>0$. As the number of samples increases, does this means we are getting samples of all $\\mathcal{S}\\times\\mathcal{A}$? So does this situation in some sense recover the generated model setting (for any policy $\\pi$, the offline data can be viewed as a generated sample)?", "limitations": "This paper is well-written, but some of the assumptions seem to be a little strong to me and hence limit the contribution of the paper. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S: This work considered the RL problems under the offline setting, which is more challenging. A global optimality is also proposed. \n\nW: (1).The main challenge in robust RL is to find the worst case, however in this paper, it is solved by assuming a fail-state. This assumption greatly reduces the difficulties of the problem hence I'm afread it is too strong. Although it can be justified in practice, it is still too strong for me. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3yD2_RqcI7", "reviewer": "Reviewer_9rit", "summary": "Offline RL has been receiving enormous attention recently due to its practical application. However, as the offline data generation method could be noisy, identifying a robust solution by accounting uncertain model parameters has great impact. The paper proposes a robust fitted Q-learning (RFQI) algorithm that used a dual formulation to work with nominal training model. The dual problem is reformulated with function optimization and solved with risk minimization techniques, which is then used to approximate robust Bellman update. Theoretical upper bounds are derived for the RFQI algorithm. Experimental results on simple toy environments demonstrate that RFQI outperforms non-robust algorithms.", "questions": "1. How is your work fundamentally different from R-LSVI?\n2. Are you planning to open-source the codes for reproducibility purposes?\n", "limitations": "The paper missed citing some recent work on offline robust RL (e.g., Zhang et. al. 2022). Experimental results are not sufficient to empirically validate the claims. Validating the performance against state-of-the-art robust offline RL methods on a set of real-world simulated environments will strengthen the paper.  ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Robust offline RL is a practically important and challenging problem. RFQI reformulated the challenging estimation problem to a function optimization model and used it to update the robust Q-function. I appreciate the theoretical results and insights provided in Theorem 1. Experiments are performed on simple gym environments which are not exhaustive enough. Furthermore, only non-robust benchmark algorithms are used to validate the performance. Comparing against something like R-LSVI (Zhang et. al. 2022) would have been great.\n\nReferences:\n1.\tZhang, Xuezhou, Yiding Chen, Xiaojin Zhu, and Wen Sun. \"Corruption-robust offline reinforcement learning.\" In International Conference on Artificial Intelligence and Statistics, pp. 5757-5773. PMLR, 2022. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept