Title: The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: distributionally robust reinforcement learning

robust Markov decision processes

sample complexity
Abstract: This paper investigates model robustness in reinforcement learning (RL) via the framework of distributionally robust Markov decision processes (RMDPs). Despite recent efforts, the sample complexity of RMDPs is much less understood regardless of the uncertainty set in use; in particular, there exist large gaps between existing upper and lower bounds, and it is unclear if distributional robustness bears any statistical implications when benchmarked against standard RL. In this paper, assuming access to a generative model, we derive the sample complexity of RMDPs---when the uncertainty set is measured via either total variation or $\chi^2$ divergence over the full range of uncertainty levels---using a model-based algorithm called distributionally robust value iteration, and develop  minimax lower bounds to benchmark its tightness. Our results not only strengthen the prior art in both directions of upper and lower bounds, but also deliver surprising messages that learning RMDPs is not necessarily easier or more difficult than standard MDPs. In the case of total variation, we establish the minimax-optimal sample complexity of RMDPs which is always smaller than that of standard MDPs. In the case of $\chi^2$ divergence, we establish the sample complexity of RMDPs that is tight up to polynomial factors of the effective horizon, and grows linearly with respect to the uncertainty level when it approaches infinity.

The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model

Abstract

This paper investigates model robustness in reinforcement learning (RL) via the framework of distributionally robust Markov decision processes (RMDPs). Despite recent efforts, the sample complexity of RMDPs is much less understood regardless of the uncertainty set in use; in particular, there exist large gaps between existing upper and lower bounds, and it is unclear if distributional robustness bears any statistical implications when benchmarked against standard RL. In this paper, assuming access to a generative model, we derive the sample complexity of RMDPs -when the uncertainty set is measured via either total variation or χ 2 divergence over the full range of uncertainty levels - using a model-based algorithm called distributionally robust value iteration, and develop minimax lower bounds to benchmark its tightness. Our results not only strengthen the prior art in both directions of upper and lower bounds, but also deliver surprising messages that learning RMDPs is not necessarily easier or more difficult than standard MDPs. In the case of total variation, we establish the minimax-optimal sample complexity of RMDPs which is always smaller than that of standard MDPs. In the case of χ 2 divergence, we establish the sample complexity of RMDPs that is tight up to polynomial factors of the effective horizon, and grows linearly with respect to the uncertainty level when it approaches infinity.

1 Introduction

Reinforcement learning (RL) deals with the problem of learning to make sequential decisions based on trial-and-error interactions with some unknown environment. As a fast-growing subfield of artificial intelligence, it has achieved significant success in a variety of domains such as large language model alignment (OpenAI, 2023; Ziegler et al., 2019), healthcare (Liu et al., 2019; Fatemi et al., 2021), robotics and control (Kober et al., 2013; Mnih et al., 2013). Due to the high dimensionality of the state-action space, achieving sample efficiency lies at the core of modern RL practice, especially in various sample-starved applications. As a result, a large portion of efforts in RL has been put in designing sample-efficient algorithms and understanding the fundamental statistical difficulty for diverse RL problems (Azar et al., 2013; Li et al., 2020).

While standard RL has been heavily invested recently, its use can be significantly hampered in practice due to the sim-to-real gap, where an agent trained in an ideal, nominal environment might be extremely sensitive and fail catastrophically when the deployed environment is subject to small changes in task objectives or unexpected perturbations (Zhang et al., 2020a; Klopp et al., 2017; Mahmood et al., 2018). Consequently, in addition to maximizing the long-term cumulative reward,

Matthieu Geist § Google

robustness becomes another critical goal for an RL agent, especially in high-stake applications such as robotics, autonomous driving, clinical trials, financial investments, and so on. To address this, distributionally robust RL (Iyengar, 2005; Nilim and El Ghaoui, 2005), which leverages insights from distributionally robust optimization and supervised learning (Rahimian and Mehrotra, 2019; Gao, 2020; Bertsimas et al., 2018; Duchi and Namkoong, 2018; Blanchet and Murthy, 2019), becomes a natural and versatile framework with the goal of learning a policy that performs well even when the deployed environment deviates from the nominal one in the face of environment uncertainty.

In this paper, we are particularly interested in understanding whether, and how, the choice of distributional robustness bears statistical implications in learning the desired policy, by studying the sample complexity in the widely-used generative model (Kearns and Singh, 1999). Suppose that one has access to a generative model which draws samples from a Markov decision processes (MDP) with a nominal transition kernel. Standard RL aims to learn the optimal policy tailored for the nominal kernel based on this set of samples, where the sample complexity has been well understood with matching upper and lower bounds developed recently (Azar et al., 2013; Li et al., 2020). In contrast, distributionally robust RL - leveraging the same set of samples - aims to learn the optimal robust policy whose worst-case performance is maximized when the transition kernel is from some prescribed uncertainty set around the nominal kernel, a setting that is referred to as the robust MDP (RMDPs). 6 Clearly, this ensures that the performance of the learned policy is robust and does not fail catastrophically as long as the sim-to-real gap is not too large. It is then natural to wonder how the robustness consideration impacts the RL performance: should we always prefer to learn a robust policy for a given set of samples? Is there a statistical premium when asking for additional robustness?

Compared with standard MDPs, RMDPs is a richer class of models since one additionally needs to prescribe the shape and size of the uncertainty set, which is usually hand-picked as a small ball around the nominal kernel measured with respect to some distance measure ρ and uncertainty level σ . To ensure the tractability of solving RMDPs, the uncertainty set is usually assumed to obey certain structures, where the uncertainty set can be decomposed as a product of independent uncertainty subsets over each state or state-action pair (Zhou et al., 2021; Wiesemann et al., 2013), denoted as the s - and ( s, a ) -rectangular rectangularity respectively; in particular, our paper adopts the second choice by assuming the uncertainty set satisfies the ( s, a ) -rectangularity. An additional challenge with RMDPs arises from distribution shift, where the transition kernel drawn from the uncertainty set can be different from the nominal kernel, leading to complicated nonlinearity and nested optimization in the problem structure not present in standard MDPs.

1.1 Prior art and open questions

In this paper, we focus on understanding the sample complexity of learning the optimal robust policy of RMDPs in the infinite-horizon setting assuming access to a generative model, when the uncertainty set is measured using one of the f -divergence: total variation (TV) distance and χ 2 divergence. These two choices are motivated by their practical appeal: easy to implement, and already adopted by empirical RL (Lee et al., 2021; Pan et al., 2023).

A popular learning approach is model-based, which first estimates the nominal transition kernel using a plug-in estimator based on the collected samples, and then runs a planning algorithm such as a robust variant of value iteration on top of the estimated RMDP. Despite the surge of recent activities, however, existing statistical guarantees for the above paradigm remain highly inadequate, as we shall elaborate momentarily (see Table 1 and Table 2 respectively for a summary). For concreteness, let S be the size of the state space, A be the size of the action space, γ be the discount factor (and the effective horizon 1 1 -γ ), and σ be the uncertainty level. We are interested in the sample complexity -the number of samples needed for an algorithm to output a policy whose robust value function (the worst-case value over all the transition kernels in the uncertainty set) is at most ε away from the optimal robust one - with respect to all salient problem parameters.

· Large gaps between existing upper and lower bounds. There remained large gaps between the sample complexity upper and lower bounds established in prior literature, regardless of the divergence metric in use. Specifically, considering the cases using either TV distance or χ 2 divergence, the state-of-the-art upper bounds (Panaganti and Kalathil, 2022) scales quadratically with the size S of the

<latexi sh

1_b

64="GVUv

Y8N

Bf

K

RJm

y

LOAZ0F

>

CDX

c

7Tg

3

+I

SH

E

jP

W

Mz

5

kn

w

d

q

9

/

p

u

o

2

Q

r

<latexi sh

1_b

64="Cfm vc9

O

7

>A

D

T

JBF

+IL

Z

YS

E

U

K

P

WyN

M

G5

kn w8

d

q

/

0

u

Q

r

<latexi sh

1_b

64="

vK

LBS2

ZdXIQ

8p

EV

+G

g

>A

C

H

c

9

wN

zF

R

oO

u

j

P

JW

5

7

3

n

U

0

mD

M

/

f

y

r

q

T

k

Y

<latexi sh

1_b

64="UG

WZX

ryT

2MB/f

AKkI

R5

o

>

C

3

c

V

7SgN

FJ

d

m

Y

p

8u

H

O

PLz

w

0

QE

9D

v

+

q

n

j

Upper & minimax lower bound

(this work)

0

O

(1)

σ

Standard MDPs

upper & minimax lower bound

Sample complexity

SA

(1

-

γ

)

3

ε

2

SA

(1

-

γ

)

2

ε

2

SA

(1

-

γ

)

ε

2

Lower bound [Yang et al.]

SA

(1

-

γ

)

σ

4

ε

2

SA

(1

-

γ

)

2

ε

2

σ

1

O

(1

-

γ

)

Upper bound

S

2

A

(1

-

γ

)

4

ε

2

Lower bound [Yang et al.]

0

σ

Sample complexity

SA

(1

-

γ

)

3

ε

2

SA

(1

-

γ

)

4

ε

2

Lower bound

(this work)

SA

(1

-

γ

)

2

ε

2

σ

Upper bound

(this work)

SA

(1

-

γ

)

ε

2

O

(1

-

γ

)

O

(1)

O

(

1

/

(1

-

γ

)

)

Standard MDPs

upper & minimax lower bound

SAσ

ε

2

SAσ

(1

-

γ

)

4

ε

2

SAσ

(1

-

γ

)

4

(1 +

σ

)

4

S

2

A

(1

-

γ

)

4

ε

2

Upper bound

S

2

Aσ

(1

-

γ

)

4

ε

2

(a) TV distance

(b)

χ

2

distance

state space, while the lower bound (Yang et al., 2022) exhibits only linear scaling with S . Moreover, in the χ 2 divergence case, the state-of-the-art upper bound grows linearly with the uncertainty level σ when σ ≳ 1 , 7 while the lower bound (Yang et al., 2022) is inversely proportional to σ . These lead to unbounded gaps between the upper and lower bounds as σ grows. Can we hope to close these gaps for RMDPs?

Benchmarking with standard MDPs. Perhaps a more pressing issue is that, past works failed to provide an affirmative answer regarding how to benchmark the sample complexity of RMDPs with that of standard MDPs regardless of the chosen shape (determined by ρ ) or size (determined by σ ) of the uncertainty set, given the large unresolved gaps mentioned above. Specifically, existing sample complexity upper (resp. lower) bounds are all larger (resp. smaller) than the sample size requirement for standard MDPs. As a consequence, it remains mostly unclear whether learning RMDPs is harder or easier than learning standard MDPs.

1.2 Main contributions

To address these questions, we have developed new upper bounds on learning RMDPs with TV distance and χ 2 distance in the infinite-horizon setting using the model-based approach called distributionally robust value iteration ( DRVI ), as well as minimax lower bounds to help gauge their

<latexi sh

1_b

64="r

PGDo38dCU

L

A

/f

n

>

B

H

c

V

NS

EJ

q

9

0m

M

F

2

p

Q

+

z

Z

j

Y

w

y

I

W

X

7

v5

k

K

T

R

O

<latexi sh

1_b

64="r

U

pSwy

/TEf

OQH

V5o

0

>A

B8n

c

DL

gN

Jz

G

M

Y

vX

Z

m

k9

W

P

CR7

qK

+

u

d

F

3

j

I

2

<latexi sh

1_b

64="F

N5+Uv

/GS

7Wym gp

TH

I

>A

C

c

VDL

BEJz

Mr

Y

Q

9R

j

k

Z

X

d

P

wo

q

O

u

2

n

0

f

K

3

8

<latexi sh

1_b

64="qdMc

Y

W+rygP

Q5

2mnz0

U

>A

CFH

VDLS

NBEJ

G

OXw

o

j

I

vZ

/

u

K

8

k

3

f

7

T

p

R

9

<latexi sh

1_b

64="vYw ro

KF

W

Cqz

9dIQ

>A

B

3

c

V

NS

M

EJ2

X7

+

T

R

pu

f

OL

D

k0

m

U

/

P

H

Z

8

5

j

g

G

y

n

<latexi sh

1_b

64="T

IgPQprnJE

HS

+

uM3

y

8

>A

B

c

V

N

U

q/

9L

0m

o

F

2

z

D

G

Z

j

Y

w

f

d

W

X

7

v5

C

k

K

R

O

<latexi sh

1_b

64="uv q5

X

k

Z/Bm

y

DEz

G8

U

>A

7

c

V

LSgN

O

r

f

Y9

oP

K

Q

j

d

C

HM

p

T

I

3

J

0F

W

+

n

w

R

2

<latexi sh

1_b

64="

vK

LBS2

ZdXIQ

8p

EV

+G

g

>A

C

H

c

9

wN

zF

R

oO

u

j

P

JW

5

7

3

n

U

0

mD

M

/

f

y

r

q

T

k

Y

<latexi sh

1_b

64="

v

r

f5S

QY

>

CDn

c

V

7

gNBF

3

d

0

O

uI

Z

X

k

P

/

H

w

p

T

j

E

K

<latexi sh

1_b

64="GVUv

Y8N

Bf

K

RJm

y

LOAZ0F

>

CDX

c

7Tg

3

+I

SH

E

jP

W

Mz

5

kn

w

d

q

9

/

p

u

o

2

Q

r

<latexi sh

1_b

64="

k

8

M

GKAu

D

>

C

3

c

V

7S

NBF

r

d

m

R

Y

Z

I

O

PLz

w

/

U

0

QE

o9

v

+

q2n

y

W

j

T

<latexi sh

1_b

64="r

PGDo38dCU

L

A

/f

n

>

B

H

c

V

NS

EJ

q

9

0m

M

F

2

p

Q

+

z

Z

j

Y

w

y

I

W

X

7

v5

k

K

T

R

O

<latexi sh

1_b

64="r

U

pSwy

/TEf

OQH

V5o

0

>A

B8n

c

DL

gN

Jz

G

M

Y

vX

Z

m

k9

W

P

CR7

qK

+

u

d

F

3

j

I

2

<latexi sh

1_b

64="kq2J+3rO

5DZjA

y

P

>

CFH

c

LS

M

IY

9V

WK

Q

d

G

v0

n

p

u

7

N

8E

f

m

U

w

R

B

o

/

<latexi sh

1_b

64="F

N5+Uv

/GS

7Wym gp

TH

I

>A

C

c

VDL

BEJz

Mr

Y

Q

9R

j

k

Z

X

d

P

wo

q

O

u

2

n

0

f

K

3

8

<latexi sh

1_b

64="vYw ro

KF

W

Cqz

9dIQ

>A

B

3

c

V

NS

M

EJ2

X7

+

T

R

pu

f

OL

D

k0

m

U

/

P

H

Z

8

5

j

g

G

y

n

<latexi sh

1_b

y

Bk vC7

H9X+oc0

>A

VDLS

N

J3

2

g

W

Tj

r

M

Z

u/

8

w

n

Q

d

p

I

P

G

O

f

m

E

<latexi sh

1_b

64="

WrwfQqjZ

8

oy

OB

U9L

>A

CFn

c

V

S

N

E

3zM

v

KX

Y

g7

2

p

R

d

T0

HPk

G

+

m

D

I

u

J

/

5

<latexi sh

1_b

64="PnoXAR

Syv

WgCjM

K+9f

FBrc

>

E3

VDL

N

J

UY

Q

u

O

7

k

Z

2

8

d/

H

0

wq

T

mp

I

5

G

z

<latexi sh

1_b

64="Sr

F

9

dgImnyz

R

opU

J

j

k

>A

CB3

c

VDL

N

2vqE

w

Z

+

0

5

/

7

Xf

M

O

8Y

K

W

G

Q

u

H

T

P

<latexi sh

1_b

64="uv q5

X

k

Z/Bm

y

DEz

G8

U

>A

7

c

V

LSgN

O

r

f

Y9

oP

K

Q

j

d

C

HM

p

T

I

3

J

0F

W

+

n

w

R

2

tightness and provide benchmarking with standard MDPs. As shall be outlined, these new analyses lead to new insights on the interplay between the geometry of uncertainty sets and the statistical sample complexity.

Sample complexity of RMDPs under the TV distance. We summarize the results and comparisons to prior works in Table 1; see Figure 1(a) for an illustration.

· Minimax-optimal sample complexity. We prove that DRVI reaches ε accuracy as soon as the sample complexity is on the order of for all σ ∈ (0 , 1) , assuming that ε is small enough. In addition, a matching minimax lower bound (modulo some logarithmic factor) is established to guarantee the tightness of the upper bound. To the best of our knowledge, this is the first minimax-optimal sample complexity for RMDPs, which was previously unavailable regardless of the divergence metric and uncertainty level in use and is over the full range of the uncertainty level.

· RMDPs are easier than standard MDPs under the TV distance. Given the sample complexity O ( SA (1 -γ ) 3 ε 2 ) of standard MDPs, it can be seen that RMDPs under the TV distance is never harder than standard MDPs, where it matches that of standard MDPs when σ ≲ 1 -γ , and becomes simpler by a factor of σ/ (1 -γ ) when 1 -γ ≲ σ < 1 . Therefore, in this case, the robustness comes almost for free since we do not need to collect more samples to reach the same accuracy.

Sample complexity of RMDPs under the χ 2 distance. We summarize the results and comparisons to prior works in Table 2; see Figure 1(b) for an illustration.

· Near-optimal sample complexity. It is established that DRVI reaches ε accuracy as soon as the sample complexity is on the order of ˜ O ( SA (1+ σ ) (1 -γ ) 4 ε 2 ) for all σ ∈ (0 , ∞ ) , which is the first sample complexity that scales linearly with respect to the size of the state space S in the infinite-horizon setting, breaking the quadratic scaling bottleneck presented in prior works (Panaganti and Kalathil, 2022; Yang et al., 2022). We have also developed a strengthened lower bound that is optimized by leveraging the geometry of the uncertainty set under different ranges of σ . Comparing the two bounds, they match at ˜ O ( SA (1 -γ ) 4 ε 2 ) when σ ≍ 1 , and have a bounded gap no larger than a polynomial factor of the effective horizon 1 / (1 -γ ) over the entire range of the uncertainty level, again significantly improving prior art that exhibits an unbounded gap.

· RMDPs can be harder than standard MDPs under the χ 2 distance. Somewhat surprisingly, the new lower bound suggests that RMDPs in this case can be much harder than standard MDPs, at least for certain ranges of the uncertainty level. We single out two regimes of particular interest. First, when σ ≍ 1 , the sample size requirement of RMDPs is ˜ O ( SA (1 -γ ) 4 ε 2 ) , which is provably harder than

standard MDPs by a factor of the effective horizon 1 1 -γ . Second, when σ ≳ 1 (1 -γ ) 3 , the lower bound exceeds the sample complexity of standard MDPs and keeps growing linearly with the uncertainty level σ .

In sum, our sample complexity bounds not only strengthen the prior art in both directions of upper and lower bounds, but also reveal new insights on how the additional consideration of distributional robustness fundamentally changes the sample complexity of RL in a surprising manner. It turns out that RMDPs are not necessarily harder nor easier than standard MDPs, but the answer is far more nuanced and highly dependent on both the size and shape of the uncertainty set, which are not elucidated in prior analyses.

2 Problem formulation

In this section, we introduce the model of distributionally robust Markov decision processes (RMDPs) focusing on the discounted infinite-horizon setting, the sampling mechanism, as well as our goal.

Standard MDPs. To begin, we first introduce the standard Markov decision process (MDP), which facilitates the understanding of RMDPs. A discounted infinite-horizon MDP is represented by M = ( S , A , P, γ, r ) , where S = { 1 , · · · , S } and A = { 1 , · · · , A } are the finite state and action spaces, respectively, γ ∈ [0 , 1) is the discount factor, P : S × A → ∆( S ) denotes the probability transition kernel, and r : S × A → [0 , 1] is the immediate reward function which is assumed to be deterministic. A policy is denoted as π : S → ∆( A ) , which specifies the action selection probability over the action space in any state. When the policy is deterministic, we overload the notation and refer to π ( s ) as the action selected by policy π in state s . To characterize the long term cumulative reward, the value function V π,P for any policy π under the transition kernel P is defined by where the expectation is taken over the randomness of the trajectory { s t , a t , r t } ∞ t =0 generated by executing policy π under the transition kernel P , namely, a t ∼ π ( s t ) , and s t +1 ∼ P ( · | s t , a t ) . Similarly, the Q-function Q π,P associated with any policy π under the transition kernel P is defined as

Distributionally robust MDPs. In this work, we focus on the discounted infinite-horizon distributionally robust MDP (RMDP), denoted as M rob = {S , A , U σ ρ ( P 0 ) , γ, r } , where S , A , γ, r are defined the same as those in the above standard MDP. A key distinction from the standard MDP, is that rather than assuming a fixed transition kernel P , it postulates the transition kernel lies in an uncertainty set U σ ρ ( P 0 ) centered around a nominal kernel P 0 : S × A → ∆( S ) , where the uncertainty set is specified using some distance metric ρ of radius σ > 0 . In particular, given the nominal transition kernel P 0 and some uncertainty level σ , the uncertainty set-with divergence ρ : ∆( S ) × ∆( S ) → R + -is specified as

In other words, the uncertainty is imposed in a separate manner for each state-action pair, obeying the so-called ( s, a ) -rectangularity (Zhou et al., 2021; Wiesemann et al., 2013). In RMDPs, we are interested in the worst-case performance of a policy π over all the possible transition kernels in the uncertainty set. This is measured by the robust value function V π,σ and the robust Q-function Q π,σ in M rob , defined respectively as

Optimal robust policy and robust Bellman operator. Generalizing standard MDPs, it is wellknown that there exists at least one deterministic policy that maximizes the robust value function and

the robust Q-function simultaneously (Iyengar, 2005; Nilim and El Ghaoui, 2005). Therefore, we denote the optimal robust value function (resp. optimal robust Q-function ) as V ⋆,σ (resp. Q ⋆,σ ), and the optimal robust policy as π ⋆ , which satisfies

The robust Bellman operator (Iyengar, 2005; Nilim and El Ghaoui, 2005) is denoted as T σ ( · ) : R SA → R SA , which is defined as follows: for all ( s, a ) ∈ S × A ,

Given that Q ⋆,σ is the unique fixed point of T σ , one can recover the optimal robust value function and Q-function using a procedure termed distributionally robust value iteration -which generalizes the standard value iteration-by recursively applying the robust Bellman operator from some fixed initialization. In addition, this procedure converges rather fast due to the nice γ -contraction property of T σ (Iyengar, 2005; Nilim and El Ghaoui, 2005) with respect to the ℓ ∞ norm.

Specification of the divergence ρ . We consider two popular choices of the uncertainty set measured in terms of the f -divergence: total variation and χ 2 divergence, given respectively by

Sampling mechanism: a generative model. Following Zhou et al. (2021); Panaganti and Kalathil (2022), we assume the access to a generative model or a simulator (Kearns and Singh, 1999), which allows us to collect N independent samples from the nominal kernel P 0 at each state-action pair:

Note that ρ TV ( P s,a , P 0 s,a ) ∈ [0 , 1] and ρ χ 2 ( P s,a , P 0 s,a ) ∈ [0 , ∞ ) in general. As we shall see, the two choices convey drastically different messages in the statistical complexity of RMDPs.

The total sample size therefore is NSA .

Goal. Given the collected samples, the task is to learn the robust optimal policy for the RMDP with some uncertainty set U σ ρ ( P 0 ) around the nominal kernel accurately using as few samples as possible. Specifically, given some accuracy level ε > 0 , the goal is to seek an ε -optimal robust policy π obeying V ⋆,σ ( s ) -V ̂ π,σ ( s ) ≤ ε for all s ∈ S .

̂ 3 Model-based algorithm: distributionally robust value iteration

We consider a model-based strategy, which first constructs an empirical nominal transition kernel based on the collected samples, and then applies distributionally robust value iteration (DRVI) to recover the optimal robust policy.

Empirical nominal kernel. The empirical nominal transition kernel ̂ P 0 ∈ R SA × S can be constructed using the empirical frequency of visits, i.e.

which leads to an empirical RMDP ̂ M rob = {S , A , U σ ρ ( ̂ P 0 ) , γ, r } . Analogously, we can define the corresponding robust value function (resp. robust Q-function) of policy π in ̂ M rob as ̂ V π,σ (resp. ̂ Q π,σ ) (cf. (5)). In addition, we denote the corresponding optimal robust policy as ̂ π ⋆ and the optimal robust value function (resp. optimal robust Q-function ) as ̂ V ⋆,σ (resp. ̂ Q ⋆,σ ) (cf. (5)), which satisfies the robust Bellman optimality equation:

Algorithm 1: Distributionally robust value iteration ( DRVI ) for infinite-horizon RMDPs.

1 input: empirical nominal transition kernel P ; reward function r ; uncertainty level σ ; number of

̂ 0 iterations T . 2 initialization: ̂ Q 0 ( s, a ) = 0 , ̂ V 0 ( s ) = 0 for all ( s, a ) ∈ S × A . 3 for t = 1 , 2 , · · · , T do 4 for s ∈ S , a ∈ A do 5 Set ̂ Q t ( s, a ) according to (13); 6 for s ∈ S do 7 Set ̂ V t ( s ) = max a ̂ Q t ( s, a ) ; 8 output: ̂ Q T , ̂ V T and ̂ π obeying ̂ π ( s ) := arg max a ̂ Q T ( s, a ) .

DRVI : distributionally robust value iteration. To solve for the fixed point of ̂ T σ , we introduce distributionally robust value iteration ( DRVI ), which is summarized in Algorithm 1. Starting from some initialization Q 0 = 0 , the update rule at the t -th ( t ≥ 1 ) step can be formulated as:

where ̂ V t -1 ( s ) = max a ̂ Q t -1 ( s, a ) for all s ∈ S . However, directly solving (13) is computationally prohibitive since it involves optimization over an S -dimensional probability simplex at each iteration, especially when the dimension of the state space S is prohibitive. Fortunately, in view of strong duality (Iyengar, 2005), (13) can be equivalently solved using its dual problem, which concerns optimizing of a scalar dual variable and thus can be solved efficiently. The specific form of the dual problem depends on the choice of the divergence ρ , which we discuss in a more detailed version.

4 Theoretical guarantees: sample complexity analyses

We now present our main results, which concern the sample complexities of learning RMDPs when the uncertainty set is specified using the TV distance or the χ 2 divergence. Surprisingly, the choice of the uncertainty set can lead to dramatic consequence in the sample size requirement.

4.1 The case of TV distance: RMDP is easier than standard MDP

We start with the case when the uncertainty set is measured via the TV distance, where Theorem 1 presents the sample complexity upper bound above which DRVI is able to find an ε -optimal robust policy in a small number of iterations; the key challenge of the analysis is to carefully control the robust value function V π,σ as a function of uncertainty level σ .

Theorem 1 (Upper bound using TV distance) . Fix the uncertainty set U σ ρ ( · ) = U σ TV ( · ) using the TV distance in (7) . Consider any discount factor γ ∈ [ 1 4 , 1 ) , uncertainty level σ ∈ (0 , 1) , and δ ∈ (0 , 1) . With probability at least 1 -δ , the output ̂ π from Algorithm 1 with at most T = C 1 log ( N (1 -γ ) ) iterations yields V ⋆,σ ( s ) -V ̂ π,σ ( s ) ≤ ε for any ε ∈ ( 0 , √ 1 / max { 1 -γ, σ } ] , as long as the total number of samples obeys NSA ≥ C 2 SA (1 -γ ) 2 max { 1 -γ,σ } ε 2 log ( SAN (1 -γ ) δ ) . Here, C 1 , C 2 > 0 are some large enough universal constants.

Before discussing the implications of Theorem 1, we present a matching minimax lower bound that confirms the optimality of the upper bound, which in turn pins down the sample complexity requirement for learning RMDPs with TV distance.

Theorem 2 (Lower bound using TV distance) . Consider any tuple ( S, A, γ, σ, ε ) obeying σ ∈ (0 , 1 -c 0 ] with 0 < c 0 ≤ 1 8 being any small enough positive constant, γ ∈ [ 1 2 , 1 ) , and ε ∈ ( 0 , c 0 256(1 -γ ) ] . We

can construct two infinite-horizon RMDPs M 0 , M 1 defined by the uncertainty set U σ ρ ( · ) = U σ TV ( · ) , an initial state distribution φ , and a dataset with N independent samples for each state-action pair over the nominal transition kernel (for M 0 and M 1 respectively), such that provided that NSA ≤ c 0 SA log 2 8192(1 -γ ) 2 max { 1 -γ,σ } ε 2 . The infimum is taken over all estimators ̂ π , and P 0 (resp. P 1 ) denotes the probability when the RMDP is M 0 (resp. M 1 ).

Below, we interpret the above theorems and highlight several key implications about the sample complexity requirements for learning RMDPs with TV distance.

Near minimax-optimal sample complexity. Theorem 1 shows that the total number of samples required for DRVI to yield ε -accuracy is

Taking together with the minimax lower bound asserted in Theorem 2, this confirms the near minimaxoptimality of the sample complexity up to some logarithmic factor almost over the full range of the uncertainty level σ , which scales linearly with respect to the size of the state-action space.

RMDPs is easier than standard MDPs with TV distance. Recall that the sample complexity requirement for standard MDP (Agarwal et al., 2020; Li et al., 2020) to yield ε accuracy is ˜ O ( SA (1 -γ ) 3 ε 2 ) . Comparing with the sample complexity requirement in (14) for RMDPs with TV distance, this confirms that the latter is at least as easy as-if not easier-than standard MDPs. In particular, when σ ≲ 1 -γ is small, the sample complexity of RMDPs is the same as the standard MDPs, which is expected since the RMDP reduces to the standard MDP when σ = 0 . On the other hand, when 1 -γ ≲ σ < 1 , the sample complexity of RMDPs becomes ˜ O ( SA (1 -γ ) 2 σε 2 ) , which is smaller than that of standard MDPs by a factor of σ/ (1 -γ ) .

Comparison with state-of-the-art bounds. For the upper bound, our results (cf. Theorem 1) significantly improves over the prior art ˜ O ( S 2 A (1 -γ ) 4 ε 2 ) of Panaganti and Kalathil (2022) by at least a factor of S 1 -γ and even S (1 -γ ) 2 when the uncertainty level 1 -γ ≲ σ < 1 is large. Turning to the lower bound side, Yang et al. (2022) developed a lower bound for RMDPs under the TV distance, which scales as ˜ O ( SA (1 -γ ) ε 2 min { 1 (1 -γ ) 4 , 1 σ 4 }) . Clearly, this is worse than ours by a factor of σ 3 (1 -γ ) 3 ∈ ( 1 , 1 (1 -γ ) 3 ) in the regime where 1 -γ ≲ σ < 1 .

4.2 The case of χ 2 distance: RMDP can be harder than standard MDP

Wenowmoveonto the case when the uncertainty set is measured via the χ 2 distance, where Theorem 3 presents the sample complexity upper bound above which DRVI is able to find an ε -optimal robust policy in a small number of iterations.

Theorem 3 (Upper bound using χ 2 distance) . Fix the uncertainty set U σ ρ ( · ) = U σ χ 2 ( · ) using the χ 2 distance in (8) . Consider any uncertainty level σ ∈ (0 , ∞ ) , and δ ∈ (0 , 1) . With probability at least 1 -δ , the output policy ̂ π from Algorithm 1 with at most T = c 1 log ( N (1 -γ )) iterations yields V ⋆,σ ( s ) -V ̂ π,σ ( s ) ≤ ε for any ε ∈ ( 0 , 1 1 -γ ] , as long as the total number of samples obeying NSA ≥ c 2 SA (1+ σ ) (1 -γ ) 4 ε 2 log ( SAN δ ) . Here, c 1 , c 2 > 0 are some large enough universal constants.

In addition, in order to gauge the tightness of Theorem 3, and understand the minimal sample complexity requirement for learning RMDPs with χ 2 divergence, we further develop a minimax lower bound as follows.

Theorem 4 (Lower bound using χ 2 divergence) . Consider any ( S, A, γ, σ, ε ) obeying γ ∈ [ 3 4 , 1) , σ ∈ (0 , ∞ ) , and

for some small universal constant c 3 > 0 . Then we can construct two infinite-horizon RMDPs M 0 , M 1 defined by the uncertainty set U σ ρ ( · ) = U σ χ 2 ( · ) , an initial state distribution ρ , and a dataset with N independent samples for each ( s, a ) pair over the nominal transition kernel (for M 0 and M 1 respectively), such that

provided that the total number of samples

for some universal constant c 4 > 0 .

We are now positioned to highlight some key implications of the above theorems about the sample complexity requirements for learning RMDPs with χ 2 divergence.

Nearly tight sample complexity. To achieve ε -accuracy for RMDPs with χ 2 distance, Theorem 3 shows that a total number of samples on the order of ˜ O ( SA (1+ σ ) (1 -γ ) 4 ε 2 ) is sufficient for DRVI . Taking it together with the minimax lower bound in Theorem 4 confirms that the sample complexity is near-optimal up to a polynomial factor of the effective horizon 1 / (1 -γ ) over the entire range of the uncertainty level σ . In particular, when σ ≍ 1 , our sample complexity ˜ O ( SA (1 -γ ) 4 ε 2 ) is tight and matches with the lower bound; when σ ≳ 1 (1 -γ ) 3 , our sample complexity correctly predicts the linear dependency with σ , suggesting that more samples are needed when one plans for larger χ 2 -based uncertainty sets.

RMDPs can be much harder than standard MDPs with χ 2 divergence. The minimax lower bound developed in Theorem 4 exhibits a surprising non-monotonic behavior of the sample size requirement over the entire range of the uncertainty level σ ∈ (0 , ∞ ) when the uncertainty set is measured via the χ 2 divergence. When σ ≲ 1 -γ , the lower bound reduces to ˜ O ( SA (1 -γ ) 3 ε 2 ) , which matches with that of standard MDPs, as σ = 0 corresponds to standard MDP. However, two additional regimes are worth calling out:

both of which are greater than that of standard MDPs, indicating learning RMDPs with χ 2 divergence can be much harder.

Comparison with state-of-the-art bounds. Our upper bound significantly improves over the prior art ˜ O ( S 2 A (1+ σ ) (1 -γ ) 4 ε 2 ) of Panaganti and Kalathil (2022) by a factor of S , and provides the first finite-sample complexity that scales linearly with respect to S for discounted infinite-horizon RMDPs, which typically exhibit more complicated statistical dependencies than the finite-horizon counterpart. On the other hand, Yang et al. (2022) established a lower bound on the order of ˜ O ( SA (1 -γ ) 2 σε 2 ) when σ ≳ 1 -γ , which is always smaller than the requirement of standard MDPs, and diminishes when σ grows. Consequently, Yang et al. (2022) does not lead to the rigorous justification that RMDPs can be harder than standard MDPs, nor the correct linear scaling of the sample size when σ grows towards infinity.

5 Other related works

Finite-sample guarantees for standard RL. There has been a considerable amount of research into non-asymptotic sample analysis of standard RL for a variety of settings; partial examples include, but are not limited to, the works via probably approximately correct (PAC) bounds for the generative model setting (Kearns and Singh, 1999; Beck and Srikant, 2012; Li et al., 2022a; Chen et al., 2020; Azar et al., 2013; Sidford et al., 2018; Agarwal et al., 2020; Li et al., 2023, 2020; Wainwright, 2019) and the offline setting (Rashidinejad et al., 2021; Xie et al., 2021; Yin et al., 2021; Shi et al., 2022; Li

et al., 2022b; Jin et al., 2021; Yan et al., 2022), as well as the online setting via both regret-based and PAC-base analyses (Jin et al., 2018; Bai et al., 2019; Li et al., 2021; Zhang et al., 2020b; Dong et al., 2019; Jin et al., 2020; Li et al., 2022a; Jafarnia-Jahromi et al., 2020; Yang et al., 2021; Woo et al., 2023).

Robustness in RL. To address the challenges of deployed environment uncertainty, an emerging line of works begin to address robustness of RL algorithms with respect to the uncertainty or perturbation over different components of MDPs - state, action, reward, and the transition kernel; see Moos et al. (2022) for a recent review. Besides the framework of distributionally robust MDPs (RMDPs) (Iyengar, 2005) adopted by this work, to promote robustness in RL, there exist various other works including but not limited to Zhang et al. (2020a, 2021); Han et al. (2022); Qiaoben et al. (2021); Sun et al. (2021); Xiong et al. (2022) investigating the robustness w.r.t. state uncertainty. Besides, Tessler et al. (2019); Tan et al. (2020) considered the robustness w.r.t. the uncertainty of the action, and Ding et al. (2023) tackles robustness against spurious correlations.

Distributionally robust RL. Rooted in the literature of distributionally robust optimization, which has primarily been investigated in the context of supervised learning (Rahimian and Mehrotra, 2019; Gao, 2020; Bertsimas et al., 2018; Duchi and Namkoong, 2018; Blanchet and Murthy, 2019), distributionally robust dynamic programming and RMDPs have attracted considerable attention recently (Wolff et al., 2012; Kaufman and Schaefer, 2013; Ho et al., 2018; Smirnova et al., 2019; Ho et al., 2021; Goyal and Grand-Clement, 2022; Derman and Mannor, 2020; Tamar et al., 2014). In the context of RMDPs, both empirical and theoretical studies have been widely conducted, although most prior theoretical analyses focus on planning with an exact knowledge of the uncertainty set (Iyengar, 2005; Xu and Mannor, 2012; Tamar et al., 2014), or are asymptotic in nature (Roy et al., 2017).

Resorting to the tools of high-dimensional statistics, various recent works begin to shift attention to understand the finite-sample performance of provable robust RL algorithms, under diverse data generating mechanisms and forms of the uncertainty set over the transition kernel. Besides the infinite-horizon setting, finite-sample complexity bounds for RMDPs with the TV distance and the χ 2 divergence are also developed for the finite-horizon setting in Xu et al. (2023); Dong et al. (2022). In addition, many other forms of uncertainty sets have been considered associated with different divergence function including but not limited to Wasserstein distance, R-contamination, KL divergence, Wang and Zou (2021); Yang et al. (2022); Panaganti and Kalathil (2022); Zhou et al. (2021); Shi and Chi (2022); Xu et al. (2023); Wang et al. (2023a); Blanchet et al. (2023); Liu et al. (2022); Wang et al. (2023c); Liang et al. (2023); Xu et al. (2023); Badrinath and Kalathil (2021); Ramesh et al. (2023); Panaganti et al. (2022); Ma et al. (2022). Moreover, various other related problems or issues have been explored such as the difference of various uncertainty types (Wang et al., 2023b), the iteration complexity of the policy-based methods (Li et al., 2022c; Kumar et al., 2023), the cases when the uncertainty level is instance-dependent small enough (Clavier et al., 2023), and regularization-based robust RL (Yang et al., 2023; Zhang et al., 2023).

6 Discussions

This work studies sample complexity bounds for learning RMDPs when the uncertainty set is measured via the TV distance and the χ 2 divergence under the generative model. Our sample complexity bounds not only strengthen the prior art in both directions of upper and lower bounds, but also reveal new insights on how the additional consideration of distributional robustness fundamentally changes the sample complexity of RL in a surprising manner. It turns out that RMDPs are not necessarily harder nor easier than standard MDPs, but the answer is far more nuanced and highly dependent on both the size and shape of the uncertainty set under consideration. These findings could help to guide the practice of RMDPs, by raising awareness that the choice of the uncertainty set not only represents a preference in robustness, but also influences the statistical complexity of the problem.

Acknowledgments and Disclosure of Funding

The work of L. Shi and Y. Chi is supported in part by the grants ONR N00014-19-1-2404, NSF CCF-2106778, DMS-2134080, and CNS-2148212. L. Shi is also gratefully supported by the Leo Finzi Memorial Fellowship, Wei Shen and Xuehong Zhang Presidential Fellowship, and Liang

Ji-Dian Graduate Fellowship at Carnegie Mellon University. The work of Y. Wei is supported in part by the the NSF grants DMS-2147546/2015447, CAREER award DMS-2143215, CCF-2106778, and the Google Research Scholar Award. The work of Y. Chen is supported in part by the Alfred P. Sloan Research Fellowship, the Google Research Scholar Award, the AFOSR grant FA9550-22-1-0198, the ONR grant N00014-22-1-2354, and the NSF grants CCF-2221009 and CCF-1907661. The authors also acknowledge Zuxin Liu and He Wang for valuable discussions.

References

Agarwal, A., Kakade, S., and Yang, L. F. (2020). Model-based reinforcement learning with a generative model is minimax optimal. In Conference on Learning Theory , pages 67-83. PMLR.

Azar, M. G., Munos, R., and Kappen, H. J. (2013). Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model. Machine learning , 91(3):325-349.

Badrinath, K. P. and Kalathil, D. (2021). Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning , pages 511-520. PMLR.

Bai, Y., Xie, T., Jiang, N., and Wang, Y .-X. (2019). Provably efficient Q-learning with low switching cost. arXiv preprint arXiv:1905.12849 .

Beck, C. L. and Srikant, R. (2012). Error bounds for constant step-size Q-learning. Systems & control letters , 61(12):1203-1208.

Bertsimas, D., Gupta, V., and Kallus, N. (2018). Data-driven robust optimization. Mathematical Programming , 167(2):235-292.

Blanchet, J., Lu, M., Zhang, T., and Zhong, H. (2023). Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. arXiv preprint arXiv:2305.09659 .

Blanchet, J. and Murthy, K. (2019). Quantifying distributional model risk via optimal transport. Mathematics of Operations Research , 44(2):565-600.

Chen, Z., Maguluri, S. T., Shakkottai, S., and Shanmugam, K. (2020). Finite-sample analysis of stochastic approximation using smooth convex envelopes. arXiv preprint arXiv:2002.00874 .

Clavier, P., Pennec, E. L., and Geist, M. (2023). Towards minimax optimality of model-based robust reinforcement learning. arXiv preprint arXiv:2302.05372v1 .

Derman, E. and Mannor, S. (2020). Distributional robustness and regularization in reinforcement learning. arXiv preprint arXiv:2003.02894 .

Ding, W., Shi, L., Chi, Y ., and Zhao, D. (2023). Seeing is not believing: Robust reinforcement learning against spurious correlation. In Thirty-seventh Conference on Neural Information Processing Systems .

Dong, J., Li, J., Wang, B., and Zhang, J. (2022). Online policy optimization for robust MDP. arXiv preprint arXiv:2209.13841 .

Dong, K., Wang, Y., Chen, X., and Wang, L. (2019). Q-learning with UCB exploration is sample efficient for infinite-horizon MDP. arXiv preprint arXiv:1901.09311 .

Duchi, J. and Namkoong, H. (2018). Learning models with uniform performance via distributionally robust optimization. arXiv preprint arXiv:1810.08750 .

Fatemi, M., Killian, T. W., Subramanian, J., and Ghassemi, M. (2021). Medical dead-ends and learning to identify high-risk states and treatments. Advances in Neural Information Processing Systems , 34:4856-4870.

Gao, R. (2020). Finite-sample guarantees for wasserstein distributionally robust optimization: Breaking the curse of dimensionality. arXiv preprint arXiv:2009.04382 .

Goyal, V. and Grand-Clement, J. (2022). Robust markov decision processes: Beyond rectangularity. Mathematics of Operations Research .

Han, S., Su, S., He, S., Han, S., Yang, H., and Miao, F. (2022). What is the solution for state adversarial multi-agent reinforcement learning? arXiv preprint arXiv:2212.02705 .

Ho, C. P., Petrik, M., and Wiesemann, W. (2018). Fast bellman updates for robust MDPs. In International Conference on Machine Learning , pages 1979-1988. PMLR.

Ho, C. P., Petrik, M., and Wiesemann, W. (2021). Partial policy iteration for l1-robust markov decision processes. Journal of Machine Learning Research , 22(275):1-46.

Iyengar, G. N. (2005). Robust dynamic programming. Mathematics of Operations Research , 30(2):257-280.

Jafarnia-Jahromi, M., Wei, C.-Y., Jain, R., and Luo, H. (2020). A model-free learning algorithm for infinite-horizon average-reward MDPs with near-optimal regret. arXiv preprint arXiv:2006.04354 .

Jin, C., Allen-Zhu, Z., Bubeck, S., and Jordan, M. I. (2018). Is Q-learning provably efficient? In Advances in Neural Information Processing Systems , pages 4863-4873.

Jin, C., Krishnamurthy, A., Simchowitz, M., and Yu, T. (2020). Reward-free exploration for reinforcement learning. In International Conference on Machine Learning , pages 4870-4879. PMLR.

Jin, Y ., Yang, Z., and Wang, Z. (2021). Is pessimism provably efficient for offline RL? In International Conference on Machine Learning , pages 5084-5096.

Kaufman, D. L. and Schaefer, A. J. (2013). Robust modified policy iteration. INFORMS Journal on Computing , 25(3):396-410.

Kearns, M. J. and Singh, S. P. (1999). Finite-sample convergence rates for Q-learning and indirect algorithms. In Advances in neural information processing systems , pages 996-1002.

Klopp, O., Lounici, K., and Tsybakov, A. B. (2017). Robust matrix completion. Probability Theory and Related Fields , 169(1-2):523-564.

Kober, J., Bagnell, J. A., and Peters, J. (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274.

Kumar, N., Derman, E., Geist, M., Levy, K., and Mannor, S. (2023). Policy gradient for s-rectangular robust markov decision processes. arXiv preprint arXiv:2301.13589 .

Lee, J., Jeon, W., Lee, B., Pineau, J., and Kim, K.-E. (2021). Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning , pages 6120-6130. PMLR.

Li, G., Cai, C., Chen, Y., Wei, Y ., and Chi, Y . (2023). Is Q-learning minimax optimal? a tight sample complexity analysis. Operations Research .

Li, G., Chi, Y ., Wei, Y ., and Chen, Y . (2022a). Minimax-optimal multi-agent rl in markov games with a generative model. Advances in Neural Information Processing Systems , 35:15353-15367.

Li, G., Shi, L., Chen, Y ., Chi, Y ., and Wei, Y . (2022b). Settling the sample complexity of model-based offline reinforcement learning. arXiv preprint arXiv:2204.05275 .

Li, G., Shi, L., Chen, Y., Gu, Y., and Chi, Y. (2021). Breaking the sample complexity barrier to regret-optimal model-free reinforcement learning. Advances in Neural Information Processing Systems , 34.

Li, G., Wei, Y., Chi, Y ., Gu, Y ., and Chen, Y . (2020). Breaking the sample size barrier in model-based reinforcement learning with a generative model. Advances in neural information processing systems , 33:12861-12872.

Li, Y., Zhao, T., and Lan, G. (2022c). First-order policy optimization for robust markov decision process. arXiv preprint arXiv:2209.10579 .

Liang, Z., Ma, X., Blanchet, J., Zhang, J., and Zhou, Z. (2023). Single-trajectory distributionally robust reinforcement learning. arXiv preprint arXiv:2301.11721 .

Liu, S., Ngiam, K. Y., and Feng, M. (2019). Deep reinforcement learning for clinical decision support: a brief survey. arXiv preprint arXiv:1907.09475 .

Liu, Z., Bai, Q., Blanchet, J., Dong, P., Xu, W., Zhou, Z., and Zhou, Z. (2022). Distributionally robust q -learning. In International Conference on Machine Learning , pages 13623-13643. PMLR.

Ma, X., Liang, Z., Blanchet, J., Liu, M., Xia, L., Zhang, J., Zhao, Q., and Zhou, Z. (2022). Distributionally robust offline reinforcement learning with linear function approximation. arXiv preprint arXiv:2209.06620 .

Mahmood, A. R., Korenkevych, D., Vasan, G., Ma, W., and Bergstra, J. (2018). Benchmarking reinforcement learning algorithms on real-world robots. In Conference on robot learning , pages 561-591. PMLR.

Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 .

Moos, J., Hansel, K., Abdulsamad, H., Stark, S., Clever, D., and Peters, J. (2022). Robust reinforcement learning: A review of foundations and recent advances. Machine Learning and Knowledge Extraction , 4(1):276-315.

Nilim, A. and El Ghaoui, L. (2005). Robust control of Markov decision processes with uncertain transition matrices. Operations Research , 53(5):780-798.

OpenAI (2023). Gpt-4 technical report.

Pan, Y., Chen, Y., and Lin, F. (2023). Adjustable robust reinforcement learning for online 3d bin packing. arXiv preprint arXiv:2310.04323 .

Panaganti, K. and Kalathil, D. (2022). Sample complexity of robust reinforcement learning with a generative model. In International Conference on Artificial Intelligence and Statistics , pages 9582-9602. PMLR.

Panaganti, K., Xu, Z., Kalathil, D., and Ghavamzadeh, M. (2022). Robust reinforcement learning using offline data. Advances in neural information processing systems , 35:32211-32224.

Qiaoben, Y., Zhou, X., Ying, C., and Zhu, J. (2021). Strategically-timed state-observation attacks on deep reinforcement learning agents. In ICML 2021 Workshop on Adversarial Machine Learning .

Rahimian, H. and Mehrotra, S. (2019). Distributionally robust optimization: A review. arXiv preprint arXiv:1908.05659 .

Ramesh, S. S., Sessa, P. G., Hu, Y., Krause, A., and Bogunovic, I. (2023). Distributionally robust model-based reinforcement learning with large state spaces. arXiv preprint arXiv:2309.02236 .

Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. (2021). Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Neural Information Processing Systems (NeurIPS) .

Roy, A., Xu, H., and Pokutta, S. (2017). Reinforcement learning under model mismatch. Advances in neural information processing systems , 30.

Shi, L. and Chi, Y. (2022). Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. arXiv preprint arXiv:2208.05767 .

Shi, L., Li, G., Wei, Y ., Chen, Y ., and Chi, Y . (2022). Pessimistic Q-learning for offline reinforcement learning: Towards optimal sample complexity. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 19967-20025. PMLR.

Sidford, A., Wang, M., Wu, X., Yang, L., and Ye, Y. (2018). Near-optimal time and sample complexities for solving Markov decision processes with a generative model. In Advances in Neural Information Processing Systems , pages 5186-5196.

Smirnova, E., Dohmatob, E., and Mary, J. (2019). Distributionally robust reinforcement learning. arXiv preprint arXiv:1902.08708 .

Sun, K., Liu, Y., Zhao, Y., Yao, H., Jui, S., and Kong, L. (2021). Exploring the training robustness of distributional reinforcement learning against noisy state observations. arXiv preprint arXiv:2109.08776 .

Tamar, A., Mannor, S., and Xu, H. (2014). Scaling up robust MDPs using function approximation. In International conference on machine learning , pages 181-189. PMLR.

Tan, K. L., Esfandiari, Y., Lee, X. Y., and Sarkar, S. (2020). Robustifying reinforcement learning agents via action space adversarial training. In 2020 American control conference (ACC) , pages 3959-3964. IEEE.

Tessler, C., Efroni, Y ., and Mannor, S. (2019). Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning , pages 6215-6224. PMLR.

Wainwright, M. J. (2019). Stochastic approximation with cone-contractive operators: Sharp ℓ ∞ -bounds for Q-learning. arXiv preprint arXiv:1905.06265 .

Wang, S., Si, N., Blanchet, J., and Zhou, Z. (2023a). A finite sample complexity bound for distribu-

tionally robust q-learning. arXiv preprint arXiv:2302.13203 .

Wang, S., Si, N., Blanchet, J., and Zhou, Z. (2023b). On the foundation of distributionally robust reinforcement learning. arXiv preprint arXiv:2311.09018 .

Wang, S., Si, N., Blanchet, J., and Zhou, Z. (2023c). Sample complexity of variance-reduced distributionally robust Q-learning. arXiv preprint arXiv:2305.18420 .

Wang, Y. and Zou, S. (2021). Online robust reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems , 34.

Wiesemann, W., Kuhn, D., and Rustem, B. (2013). Robust markov decision processes. Mathematics of Operations Research , 38(1):153-183.

Wolff, E. M., Topcu, U., and Murray, R. M. (2012). Robust control of uncertain markov decision processes with temporal logic specifications. In 2012 IEEE 51st IEEE Conference on Decision and Control (CDC) , pages 3372-3379. IEEE.

Woo, J., Joshi, G., and Chi, Y . (2023). The blessing of heterogeneity in federated Q-learning: Linear speedup and beyond. arXiv preprint arXiv:2305.10697 .

Xie, T., Jiang, N., Wang, H., Xiong, C., and Bai, Y. (2021). Policy finetuning: Bridging sampleefficient offline and online reinforcement learning. Advances in neural information processing systems , 34.

Xiong, Z., Eappen, J., Zhu, H., and Jagannathan, S. (2022). Defending observation attacks in deep reinforcement learning via detection and denoising. arXiv preprint arXiv:2206.07188 .

Xu, H. and Mannor, S. (2012). Distributionally robust Markov decision processes. Mathematics of Operations Research , 37(2):288-300.

Xu, Z., Panaganti, K., and Kalathil, D. (2023). Improved sample complexity bounds for distributionally robust reinforcement learning. arXiv preprint arXiv:2303.02783 .

Yan, Y., Li, G., Chen, Y., and Fan, J. (2022). The efficacy of pessimism in asynchronous Q-learning. arXiv preprint arXiv:2203.07368 .

Yang, K., Yang, L., and Du, S. (2021). Q-learning with logarithmic regret. In International Conference on Artificial Intelligence and Statistics , pages 1576-1584. PMLR.

Yang, W., Wang, H., Kozuno, T., Jordan, S. M., and Zhang, Z. (2023). Avoiding model estimation in robust markov decision processes with a generative model. arXiv preprint arXiv:2302.01248 .

Yang, W., Zhang, L., and Zhang, Z. (2022). Toward theoretical understandings of robust Markov decision processes: Sample complexity and asymptotics. The Annals of Statistics , 50(6):32233248.

Yin, M., Bai, Y., and Wang, Y.-X. (2021). Near-optimal offline reinforcement learning via double variance reduction. arXiv preprint arXiv:2102.01748 .

Zhang, H., Chen, H., Boning, D., and Hsieh, C.-J. (2021). Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:2101.08452 .

Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., and Hsieh, C.-J. (2020a). Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems , 33:21024-21037.

Zhang, R., Hu, Y., and Li, N. (2023). Regularized robust mdps and risk-sensitive mdps: Equivalence, policy gradient, and sample complexity. arXiv preprint arXiv:2306.11626 .

Zhang, Z., Zhou, Y., and Ji, X. (2020b). Almost optimal model-free reinforcement learning via reference-advantage decomposition. Advances in Neural Information Processing Systems , 33.

Zhou, Z., Bai, Q., Zhou, Z., Qiu, L., Blanchet, J., and Glynn, P. (2021). Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 3331-3339. PMLR.

Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019). Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 .

Peer reviews:
comments: SUMMARY:
This paper studies the statistical nature of distributionally robust reinforcement learning under the generative model. Specifically, it studies two divergences, total variation or chi-square divergence, over the full range of uncertainty levels. The paper improves the upper and lower bounds, especially when the uncertainty levels are small. Based on these results, the paper partially answers the question of whether distributionally robust RL is harder to learn compared to the non-robust counterpart.

STRENGTHS:
The paper's theoretical results are both interesting and important in understanding the statistical limits of distributionally robust RL. The paper gives a complete story in the TV divergence and chi-square divergence.

WEAKNESSES:
A brief explanation of the reason for the improvements in both the upper and lower bounds, compared to the previous technique, would significantly help readers better understand the technical contributions of this paper.

QUESTIONS:
Can the authors briefly explain the current challenges in giving parallel results under the KL divergence?

LIMITATIONS:
See weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

comments: SUMMARY:
1. This paper studies the model robustness in RL via the framework of distributionally robust MDP.  
2. The authors derive the sample complexity of RMDPs using a model-based algorithm called distributionally robust value iteration when the uncertainty set is measured via either total variation or chi square divergence. 
3. The lower bounds are developed to benchmark its tightness
4. A interesting insight from this paper is that the required sample complexity of RMDPs can be higher or lower than the standard MDP based on the choice of uncertainty measurement.

STRENGTHS:
1. Nolvety: the improvement of the upper and lower bounds over the existing work is not easy, new techniques are developed.
2. Presentation is very clear, especially the summarizations in Tables 1-2 and the illustrations in Figure 1
3. The insights from theoretical results are interesting: RMDPs can be harder than standard MDPs under the chi square distance but can be easier than standard MDPs under the TV distance.

WEAKNESSES:
1. It is better to add experiments to empirically demonstrate the theoretical results in the paper.
2. It is better to add a discussion on how the theoretical results in this paper could help the practitioners.

QUESTIONS:
What is the intuition / theoretical explanation that whether RMDPs are not harder nor easier than standard MDPs highly depends on both the size and shape of the uncertainty set?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe research primarily focuses on the robust Markov Decision Process setting, where the objective is to learn a robust policy with the uncertainty set being measured by f-divergences using a model-based algorithm. One of the key idea of the paper lies in deriving precise sample complexity bounds for the same with access to the generative model and shows that RMDPs is not necessarily more difficult than standard MDPs, which is an important finding in general.\n\nSTRENGTHS:\nOne of the primary challenges in the field of Robust MDPs in general lies in the selection of the uncertainty set and shape and size of the uncertainty set, and the past literature has assumed certain structures around the same specifically (s,a) rectangularity to solve the robust MDP problem.  The paper provides a detailed study of the effect of uncertainty set on the robust MDP and the sample complexity analysis. A primary contribution of the paper lies in reducing the significant gap between the upper and lower bounds of past literature with different f-divergences. A standard formulation in robust MDP deals with defining the uncertainty-set $U_{\\rho}^{\\sigma}(P^0)$ with a divergence $\\rho$ and radius $\\sigma$ around the nominal transition kernel following the (s,a) rectangularity condition and solving the robust Value iteration. Although the robust value iteration is hard to solve in general, the dual of the problem is reasonably simple and can be applied due to the strong duality condition. With this, the authors develop distributionally robust value iteration and improved the upper bounds for RMDPs with various f-divergences. The Theorems also highlight the sample complexity over the entire range of the uncertainty level and how the geometry of the uncertainty set can affect the sample complexity analysis of RMDPs. Overall the paper is clearly written, with detailed analysis, and also provides the theory and proofs of earlier results as required, which I found very helpful.\n\nWEAKNESSES:\n1. In the statement of Theorem 1, where the relation of RMDP's sample complexity with the standard MDP is discussed, it states when $\\sigma \\leq 1 -\\gamma$, the sample complexity is almost similar to MDP which makes sense since for smaller $\\sigma$ means eventually we are optimizing just over the nominal $P^0$. However, the other result shows that when $\\sigma > 1- \\gamma$, the sample complexity of RMDPs becomes smaller than MDP, which is not very intuitive. When the uncertainty set over which the robust optimization is performed, when we increase the size of the set ideally the complexity should increase. Hence, it is not very clear what is the intuition of this result and is not explicitly stated here. Why the shape of the uncertainty set with TV can produce a better complexity analysis is not absolutely clear and intuitive.\n2. The authors propose a distributionally robust value iteration algorithm (Algorithm 1) which is resulting in an improved analysis. However, the novelty of the proposed algorithm is not very clear as robust value iteration and solving the robust problem with Wasserstein divergence have been there long before. It will be critical to specifically highlight the novel aspects of the algorithm and which component is causing the improvement in the analysis.\n3. The papers lack any experimental study to verify the theoretical claims for the proposed algorithm. It will be helpful if the authors can provide simple experimental evaluations or ablations on some vanilla RL environments, and how the proposed algorithm is achieving the improvements over prior in terms of sample complexity. That will make the intuition of the improvement in the algorithm very clear.\n4. It is mentioned the distribution shift issue for RMDPs when the transition kernel drawn from the uncertainty set can be different from the nominal kernel. However, it's not clear whether it is addressed by the current algorithm and how? Also, won't this issue be increased when the size of the uncertainty set is large requiring more samples, which seems opposite to what it is shown in Theorem 1? It also needs further clarity.\n\nQUESTIONS:\n1. In Theorem 1, when $\\sigma > 1- \\gamma$, the sample complexity of RMDPs becomes smaller than MDP. However, when we increase the size of the uncertainty set, indicates we are more unsure of the transition and can come from a larger radius around the nominal. Shouldn't we need more samples in this case, why is considering the total variation causing the reverse? This is critical and will be helpful to have a detailed discussion.\n2. The author compares the sample-complexity results of RMDPs with standard MDP which is interesting and important. However, the result(Th 1) holds for $\\epsilon \\in [0, \\frac{k_0}{1 - \\gamma}$. However, in Agarwal et.al, they also show the sample complexity is $O(S^2 A)$ when $\\epsilon^2 \\geq \\frac{1}{k. (1-\\gamma)^3}$ i.e. in regimes where no meaningfully accurate approximation to the actual transition probabilities can be constructed. It will be in\n3. The authors have mentioned the geometry of the uncertainty set in connection to $\\sigma$, but how is the geometry defined and what the geometry of the set refers to is not explicitly mentioned. What exactly does the geometry of the set refer to? Does it just refer to the shape owing to the type of divergence and size corresponding to the radius? Or there are further detailed connections drawn to the geometry?\n4. Although the research closes the gap b/w the upper and lower bounds with more precise bounds, the generative model assumption, in general, is restrictive, and one of the major hardness for model-based RL is to approximate the transition model which is hard.\n\n#Minor Comment\nIn several places, the citation and references were not coming properly, especially in the Appendix (Lines 663, 689, 692, 701, 800, 801, 1079), which makes it a little unsmooth to follow.\n\nLIMITATIONS:\nSome of the future and potential research directions are mentioned in the discussion, which I feel is sufficient.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents new upper and lower bounds for distributionally robust MDPs where the uncertainty set for the transition kernel is specified as a ball with $\\chi^{2}$ divergence or total variation as the distance measure. The bounds significantly improve previous results.\n\nSTRENGTHS:\nThe paper is generally well-written, and is easy and interesting to read. The results significantly improve existing bounds and are surprising. However, I didn't read the proofs.\n\nWEAKNESSES:\n* The notations $\\lesssim$ and $\\asymp$ should be defined. While $\\lesssim$ is often used in many places, it seems it should be just $\\le$ or $<$ according to the theorems.\n* The distributionally robust value iteration algorithm is taken from (Yang et al., 2022; Iyengar, 2005), but Section 3 describes the algorithm as if this is a contribution from this paper.\n* The bounds usually do not hold for all $\\gamma \\in (0, 1)$, thus there are still some gaps in the results, while the paper did not make this clear.  An intuitive discussion on why the results do not hold for $\\gamma$ values outside the ranges given in the theorems are helpful.\n* I find the statements of Theorem 2 and Theorem 4 unclear. A sample complexity lower bound result should show that no algorithm can solve a problem given fewer than a certain number of samples. However, these theorems do not mention algorithms, but instead mention two robust MDPs and take infimum with respect to the random policy $\\hat{\\pi}$. Did I miss something?\n* I find the lower bound in Theorem 4 very surprising, as the lower bound is discontinuous and non-monotonic. I note that the paper claim the bounds are at least nearly tight, but is it possible that the lower bounds are not sufficiently tight? Can an intuitive explanation for the discontinuity and non-monotonicity be given?\n* There are some discussions on whether uncertainty of the transition kernel makes learning a $\\epsilon$-optimal value function harder, and the discussion is based on comparison of sample complexity lower bounds for robust MDPs and standard MDPs. However, this may be misleading as the lower bounds can be underestimating the complexity of a problem.\n* The appendix has some broken references (??).\n\nQUESTIONS:\nI would appreciate responses to the points regarding the bounds in Weaknesses.\n\nLIMITATIONS:\nThe paper didn't discuss the limitations. I believe the paper should at least point out that the results do not hold for all $\\gamma$ values and provide a discussion on this.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the sample complexity of learning robust MDPs under generative model access, with uncertainty sets measured with  TV and chi-squared divergences. The paper proposes tight upper and lower bounds for sample complexity under TV robustness and chi-squared robustness (under some range of the uncertainty radius). Curiously, the minimax bound is smaller than standard MDPs. The upper bound is attained by a model-based distributionally robust value iteration algorithm.\n\n(changed score from 4 -> 6 during discussion.)\n\nSTRENGTHS:\n1. The paper provides tight bounds for the TV case, and somewhat tight bounds for the chi-squared case, and illustrates the interesting phenomenon where the minimax optimal bounds are sometimes easier/harder than standard RL.\n\nWEAKNESSES:\n1. The presentation can be improved. For example, the algorithm box 1 is very bare and does not contain all the necessary details for implementation. Also, computational efficiency is a key issue in robust MDPs and the current version brushed it aside (in Line 221) into the appendix, which is 46 pages. It would be better to present it clearly in the main paper. Also, what makes this algorithm different from Yang et al, 2022? Is it just an improved analysis or something fundamentally changed in the algorithm to obtain minimax-optimal bounds? \n2. The robust VI algorithm seems not novel, as it is a common method for robust MDPs, eg. Iyengar 2005.\n3. While minimax-optimal theoretical results are nice, there is limited discussion on the practicality of the algorithm, besides its worst-case sample complexities.\n\nQUESTIONS:\n1. Which parts of the current analysis breaks when trying to extend to more general MDPs, like linear MDPs? \n2. I really like Figure 1, but I do not understand the intuition on why robust MDP is easier/harder in certain cases. I can certainly follow the algebra based on minimax optimal bounds, but can you describe at an intuitive level why this \"curious price\" appears?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-21"}