Title: Connected Superlevel Set in (Deep) Reinforcement Learning and its Application to Minimax Theorems
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement learning

superlevel sets

minimax optimization

robust reinforcement learning
Abstract: The aim of this paper is to improve the understanding of the optimization landscape for policy optimization problems in reinforcement learning. Specifically, we show that the superlevel set of the objective function with respect to the policy parameter is always a connected set both in the tabular setting and under policies represented by a class of neural networks. In addition, we show that the optimization objective as a function of the policy parameter and reward satisfies a stronger “equiconnectedness” property. To our best knowledge, these are novel and previously unknown discoveries.

We present an application of the connectedness of these superlevel sets to the derivation of minimax theorems for robust reinforcement learning. We show that any minimax optimization program which is convex on one side and is equiconnected on the other side observes the minimax equality (i.e. has a Nash equilibrium). We find that this exact structure is exhibited by an interesting class of robust reinforcement learning problems under an adversarial reward attack, and the validity of its minimax equality immediately follows. This is the first time such a result is established in the literature.

Connected Superlevel Set in (Deep) Reinforcement Learning and its Application to Minimax Theorems

Abstract

The aim of this paper is to improve the understanding of the optimization landscape for policy optimization problems in reinforcement learning. Specifically, we show that the superlevel set of the objective function with respect to the policy parameter is always a connected set both in the tabular setting and under policies represented by a class of neural networks. In addition, we show that the optimization objective as a function of the policy parameter and reward satisfies a stronger 'equiconnectedness' property. To our best knowledge, these are novel and previously unknown discoveries.

Wepresent an application of the connectedness of these superlevel sets to the derivation of minimax theorems for robust reinforcement learning. We show that any minimax optimization program which is convex on one side and is equiconnected on the other side observes the minimax equality (i.e. has a Nash equilibrium). We find that this exact structure is exhibited by an interesting class of robust reinforcement learning problems under an adversarial reward attack, and the validity of its minimax equality immediately follows. This is the first time such a result is established in the literature.

1 Introduction

Policy optimization problems in reinforcement learning (RL) are usually formulated as the maximization of a non-concave objective function over a convex constraint set. Such non-convex programs are generally difficult to solve globally, as gradient-based optimization algorithms can be trapped in sub-optimal first-order stationary points. Interestingly, recent advances in RL theory [Fazel et al., 2018, Agarwal et al., 2021, Mei et al., 2020] have discovered a 'gradient domination' structure in the optimization landscape, which qualitatively means that every stationary point of the objective function is globally optimal. An important consequence of this condition is that any first-order algorithm that converges to a stationary point is guaranteed to find the global optimality.

In this work, our aim is to enhance the understanding of the optimization landscape in RL beyond the gradient domination condition. Inspired by Mohammadi et al. [2021], Fatkhullin and Polyak [2021] that discuss properties of the sublevel set for the linear-quadratic regulator (LQR), we study

the superlevel set of the policy optimization objective under a Markov decision process (MDP) framework and prove that it is always connected.

As an immediate consequence, we show that any minimax optimization program which is convex on one side and is an RL objective on the other side observes the minimax equality. We apply this result to derive an interesting and previously unknown minimax theorem for robust RL. We also note that it is unclear at the moment, but certainly possible, that the result on connected superlevel sets may be exploited to design more efficient and reliable policy optimization algorithms in the future.

1.1 Main Contribution

Our first contribution in this work is to show that the superlevel set of the policy optimization problem in RL is always connected under a tabular policy representation. We then extend this result to the deep reinforcement learning setting, where the policy is represented by a class of over-parameterized neural networks. We show that the superlevel set of the underlying objective function with respect to the policy parameters (i.e. weights of the neural networks) is connected at all levels. We further prove that the policy optimization objective as a function of the policy parameter and reward is 'equiconnected', which is a stronger result that we will define and introduce later in the paper. To the best of our knowledge, our paper is the first to rigorously investigate the connectedness of the superlevel sets for the MDP policy optimization program, both in the tabular case and with a neural network policy class.

As a downstream application, we discuss how our main results can be used to derive a minimax theorem for a class of robust RL problems. We consider the scenario where an adversary strategically modifies the reward function to trick the learning agent. Aware of the attack, the learning agent defends against the poisoned reward by solving a minimax optimization program. The formulation for this problem is proposed and considered in Banihashem et al. [2021], Rakhsha et al. [2020]. However, as a fundamental question, the validity of the minimax theorem (or equivalently, the existence of a Nash equilibrium) is still unknown. We fill in this gap by establishing the minimax theorem as a simple consequence of the equiconnectedness of the policy optimization objective.

1.2 Related Works

Our paper is closely connected to the existing works that study the structure of policy optimization problems in RL, especially those on the gradient domination condition. Our result also relates to the literature on minimax optimization for various function classes and robust RL. We discuss the recent advances in these domains to give context to our contributions.

Gradient Domination Condition. The policy optimization problem in RL is non-convex but obeys the special 'gradient domination' structure, which has been widely used as a tool to show the convergence of various gradient-based algorithms to the globally optimal policy [Agarwal et al., 2020, Mei et al., 2020, Bhandari and Russo, 2021, Zeng et al., 2021a, Xiao, 2022]. In the settings of LQR [Fazel et al., 2018, Yang et al., 2019] and entropy-regularized MDP [Mei et al., 2020, Cen et al., 2022, Zeng et al., 2022], the gradient domination structure can be mathematically described by the Polyak-Łojasiewicz (PŁ) condition, which bears a resemblance to strong convexity but does not even imply convexity. It is known that functions observing this condition can be optimized globally and efficiently by (stochastic) optimization algorithms [Karimi et al., 2016, Zeng et al., 2021b, Gower et al., 2021]. When the policy optimization problem under a standard, non-regularized MDP is considered, the gradient domination structure is weaker than the PŁ condition but still takes the form of upper bounding a global optimality gap by a measure of the magnitude of the gradient [Bhandari and Russo, 2019, Agarwal et al., 2020, 2021]. In all scenarios, the gradient domination structure prevents any stationary point from being sub-optimal.

It may be tempting to think that the gradient domination condition and the connectedness of the superlevel sets are strongly connected notions or may even imply one another. For 1-dimensional function ( f : R n → R with n = 1 ), it is easy to verify that the gradient domination condition necessarily implies the connectedness of the superlevel sets. However, when n ≥ 2 this is no longer true. In general, the gradient domination condition neither implies nor is implied by the connectedness of superlevel sets, which we illustrate with examples in Section 1.3. These two structural properties are distinct concepts that characterize the optimization landscape from different angles. This observation precludes the possibility of deriving the connectedness of the superlevel

sets in RL simply from the existing results on the gradient domination condition, and suggests that a tailored analysis is required.

Minimax Optimization & Minimax Theorems. Consider a function f : X × Y → R on convex sets X , Y . In general, the minimax inequality always holds

The seminal work Neumann [1928] shows that this inequality holds as an equality for matrix games where X ⊆ R m , Y ⊆ R n are probability simplexes and we have f ( x, y ) = x ⊤ Ay given a payoff matrix A ∈ R m × n . The result later gets generalized to the setting where X , Y are compact sets, f ( x, · ) is quasi-convex for all x ∈ X , and f ( · , y ) is quasi-concave for all y ∈ Y [Fan, 1953, Sion, 1958]. Much more recently, Yang et al. [2020] establishes the minimax equality when f satisfies the two-sided PŁ condition. For arbitrary functions f , the minimax equality need not be valid.

The validity of the minimax equality is essentially equivalent to the existence of a global Nash equilibrium ( x ⋆ , y ⋆ ) such that

The Nash equilibrium ( x ⋆ , y ⋆ ) is a point where neither player can improve their objective function value by changing its strategy. In general nonconvex-nonconcave settings where the global Nash equilibrium may not exist, alternative approximate local/global optimality notions are considered [Daskalakis and Panageas, 2018, Nouiehed et al., 2019, Adolphs et al., 2019, Jin et al., 2020].

Robust Reinforcement Learning. Robust RL studies finding the optimal policy in the worst-case scenario under environment uncertainty and/or possible adversarial attacks. Various robust RL models have been considered in the existing literature, such as: 1) the learning agent operates under uncertainty in the transition probability kernel [Goyal and Grand-Clement, 2022, Li et al., 2022, Panaganti and Kalathil, 2022, Wang et al., 2023], 2) an adversary exists and plays a two-player zero-sum Markov game against the learning agent [Pinto et al., 2017, Tessler et al., 2019], 3) the adversary does not affect the state transition but may manipulate the state observation [Havens et al., 2018, Zhang et al., 2020], 4) there is uncertainty or attack only on the reward [Wang et al., 2020, Banihashem et al., 2021, Sarkar et al., 2022], 5) the learning agent defends against attacks from a population of adversaries rather than a single one [Vinitsky et al., 2020]. A particular attack and defense model considered later in our paper is adapted from Banihashem et al. [2021].

Other Works on Connected Level Sets in Machine Learning. Last but not least, we note that our paper is related to the works that study the connectedness of the sublevel sets for the LQR optimization problem [Fatkhullin and Polyak, 2021] and for deep supervised learning under a regression loss [Nguyen, 2019]. The neural network architecture considered in our paper is inspired by and similar to the one in Nguyen [2019]. However, our result and analysis on deep RL are novel and significantly more challenging to establish, since 1) the underlying loss function in Nguyen [2019] is convex, while ours is a non-convex policy optimization objective, 2) the analysis of Nguyen [2019] relies critically on the assumption that the activation functions are uniquely invertible, while we use a non-uniquely invertible softmax activation function to generate policies within the probability simplex.

1.3 Connection between Gradient Domination and Connected Superlevel Sets

We loosely use the term 'gradient domination' to indicate that a differentiable function does not have any sub-optimal stationary points. In this section, we use two examples to show that the gradient domination condition in general does not imply or get implied by the connectedness of the superlevel sets. The first example is a function that observes the gradient domination condition but has a disconnected set of maximizers (which implies that the superlevel is not always connected).

It is obvious that the function is symmetric along the line x = 0 and that f 1 (0 , y ) = f 2 (0 , y ) for all y ∈ [ -2 , 0] . Computing the derivatives of f 1 and f 2 with respect to x , we have

We can again verify ∇ x f 1 (0 , y ) = ∇ x f 2 (0 , y ) for all y , which implies that the function f is everywhere continuous and differentiable. Visualization of f in Fig. 1 along with simple calculation (solving the system of equations ∇ x f ( x, y ) = 0 and ∇ y f ( x, y ) = 0 ) show that there are only two stationary points of f on [ -4 , 4] × [ -2 , 0] . The two stationary points are (3 . 05 , -1 . 12) and ( -3 . 05 , -1 . 12) , and they are both global maximizers on this domain, which means that the gradient domination condition is observed. However, the set of maximizers { (3 . 05 , -1 . 12) , ( -3 . 05 , -1 . 12) } is clearly disconnected.

We next present a function that has connected superlevel sets at all level but does not observe the gradient domination condition (i.e. has sub-optimal stationary points).

-5.0

2

-7.5

0

g(x,y)

-4

15.0

17.5

6

-20.0

-8

10

-4-3-2-1

-1.00

0.75

0.50

-0.25

0.00

-1.5 _1.0 0.5

0.0,

0.5

1.0

1.5

0

1

2.00

1.75

1.50

1.25

0.0

0.5

1.0

1.5

1.5

-1.0

0.5

Consider g : R 2 → R defined as

This is a volcano-shaped function, which we visualize in Fig. 1. It is obvious the superlevel set { ( x, y ) : g ( x, y ) ≥ λ } is always either a 2D circle (convex set) or a donut-shaped connected set depending on the choice of λ . However, the gradient domination condition does not hold as (0 , 0) is a first-order stationary point but not a global maximizer (it is actually a local minimizer).

Outline of the paper. The rest of the paper is organized as follows. In Section 2, we discuss the policy optimization problem in the tabular setting and establish the connectedness of the superlevel sets. Section 3 generalizes the result to a class of policies represented by over-parameterized neural networks. We introduce the structure of the neural network and the definition of super level sets in this context, and present our theoretical result. In Section 4, we use our main results on superlevel sets to derive two minimax theorems for robust RL. Finally, we conclude in Section 5 with remarks on future directions.

2 Connected Superlevel Set Under Tabular Policy

We consider the infinite-horizon average-reward MDP characterized by M = ( S , A , P , r ) . We use S and A to denote the state and action spaces, which we assume are finite. The transition probability kernel is denoted by P : S × A → ∆ S , where ∆ S denotes the probability simplex over S . The reward function r : S × A → [0 , U r ] is bounded for some positive constant U r and can also be regarded as a vector in R |S|×|A| . We use P π ∈ R S×S to represent the state transition probability matrix under policy π ∈ ∆ S A , where ∆ S A is the collection of probability simplexes over A across the state space

We consider the following ergodicity assumption in the rest of the paper, which is commonly made in the RL literature [Wang, 2017, Wei et al., 2020, Wu et al., 2020].

Assumption 1 Given any policy π , the Markov chain formed under the transition probability matrix P π is ergodic, i.e. irreducible and aperiodic.

Let µ π ∈ ∆ S denote the stationary distribution of the states induced by policy π . As a consequence of Assumption 1, the stationary distribution µ π is unique and uniformly bounded away from 0 under any π . In addition, µ π is the unique eigenvector of P π with the associated eigenvalue equal to 1 , i.e. µ π = P π µ π . Let ̂ µ π ∈ ∆ S×A denote the state-action stationary distribution induced by π , which can be expressed as

We measure the performance of a policy π under reward function r by the average cumulative reward J r ( π )

The objective of the policy optimization problem is to find the policy π that maximizes the average cumulative reward

The superlevel set of J r is the set of policies that achieve a value function greater than or equal to a specified level. Formally, given λ ∈ R , the λ -superlevel set (or superlevel set) under reward r is defined as

The main focus of this section is to study the connectedness of this set U λ,r , which requires us to formally define a connected set.

Definition 1 A set U is connected if for any x, y ∈ U there exists a continuous map p : [0 , 1] →U such that p (0) = x and p (1) = y .

We say that a function is connected if its superlevel sets are connected at all levels. We also introduce the definition of equiconnected functions.

Definition 2 Given two spaces X and Y , the collection of functions { f y : X → R } y ∈Y is said to be equiconnected if for every x 1 , x 2 ∈ X , there exists a continuous path map p : [0 , 1] →X such that

for all α ∈ [0 , 1] and y ∈ Y .

Conceptually, the collection of functions { f y : X → R } y ∈Y being equiconnected requires 1) that f y ( · ) is a connected function for all y ∈ Y (or equivalently, the set { x ∈ X : f y ( x ) ≥ λ } is connected for all λ ∈ R and y ∈ Y ) and 2) that the path map constructed to prove the connectedness of { x ∈ X : f y ( x ) ≥ λ } is independent of y .

We now present our first main result of the paper, which states that the superlevel set U λ,r is always connected.

Theorem 1 Under Assumption 1, the superlevel set U λ,r is connected for any λ ∈ R and r ∈ R |S||A| . In addition, the collection of functions { J r ( · ) : ∆ S A → R } r ∈ R |S|×|A| is equiconnected.

Our result here extends easily to the infinite-horizon discounted-reward setting since a discountedreward MDP can be regarded as an average-reward one with a slightly modified transition kernel [Konda, 2002].

The claim in Theorem 1 on the equiconnectedness of { J r } r ∈ R |S|×|A| is a slightly stronger result than the connectedness of U λ,r , and plays an important role in the application to minimax theorems discussed later in Section 4.

We note that the proof, presented in Section A.1 of the appendix, mainly leverages the fact that the value function J r ( π ) is linear in the state-action stationary distribution ̂ µ π and that there is a special connection (though nonlinear and nonconvex) between ̂ µ π and the policy π , which we take advantage

of to construct the continuous path map for the analysis. Specifically, given two policies π 1 , π 2 with J r ( π 1 ) , J r ( π 2 ) ≥ λ , we show that the policy π α defined as

is guaranteed to achieve J r ( π α ) ≥ λ for all α ∈ [0 , 1] .

Besides playing a key role in the proof of Theorem 1, our construction of this path map may inform the design of algorithms in the future. Given any two policies with a certain guaranteed performance, we can generate a continuum of policies at least as good. As a consequence, if we find two optimal policies (possibly by gradient descent from different initializations) we can generate a range of interpolating optimal policies. If the agent has a preference over these policy (for example, to minimize certain energy like in H 1 control, or if some policies are easier to implement physically), then the selection can be made on the continuum of optimal policies, which eventually leads to a more preferred policy.

3 Connected Superlevel Set Under Neural Network Parameterized Policy

In real-world reinforcement learning applications, it is common to use a deep neural network to parameterize the policy [Silver et al., 2016, Arulkumaran et al., 2017]. In this section, we consider the policy optimization problem under a special class of policies represented by an over-parameterized neural network and show that this problem still enjoys the important structure - the connectedness of the superlevel sets - despite the presence of the highly complex function approximation. Illustrated in Fig. 2, the neural network parameterizes the policy in a very natural manner which matches how neural networks are actually used in practice.

feature

neural network

policydistribution

states

vector (p(s)

parameterizedby

overactionspace

θ=collectionofallweights

Tg(· Is)

generateoutput

(softmax

0

function applied

0.7

abstractedby

1

input to

on last layer)

0.1

1

0.05

0.15

0

Mathematically, the parameterization can be described as follows. Each state s ∈ S is associated with a feature vector ϕ ( s ) ∈ R d , which in practice is usually carefully selected to summarize the key information of the state. For state identifiability, we assume that the feature vector of each state is unique, i.e.

̸

̸

To map a feature vector ϕ ( s ) to a policy distribution over state s , we employ a L -layer neural network, which in the k th layer has weight matrix W k ∈ R n k -1 × n k and bias vector b k ∈ R n k with n 0 = d and n L = |A| . For the simplicity of notation, we use Ω k to denote the space of weight and bias parameters ( W k , b k ) of layer k , and we write Ω = Ω 1 ×··· × Ω L . We use θ to denote the collection of the weights and biases

We use the same activation function for layers 1 through L -1 , denoted by σ : R → R , applied in an element-wise fashion to vectors. To ensure that the output of the neural network is a valid probability distribution, the activation function for the last layer is a softmax function, denoted by ψ : R |A| → ∆ A , i.e. for any vector v ∈ R |A|

With v ∈ R d as the input to a neural network with parameters θ , we use f θ k ( v ) ∈ R n k to denote the output of the network at layer k . For k = 1 , · · · , L , f θ k ( v ) is computed as

The policy π θ ∈ R |S|×|A| parametrized by θ is the output of the final layer:

Our analysis relies two assumptions about the structure of the neural network. The first concerns the invertibility of σ ( · ) as well as the continuity and uniqueness of its inverse, which can be guaranteed by the following:

̸

Assumption 2 σ is strictly monotonic and σ ( R ) = R . In addition, there do not exist non-zero scalars { p i , q i } i m =1 with q i = q j , ∀ i = j such that for some m> 0 , σ ( x ) = ∑ m i =1 p i σ ( x -q i ) , ∀ x ∈ R .

̸

We note that this assumption holds for common activation functions including leaky-ReLU and parametric ReLU [Xu et al., 2015].

Our second assumption is that the neural network is sufficiently over-parameterized and that the number of parameters decreases with each layer.

Assumption 3 The output of the first layer is wider than 2 |S| , and the width of the network decreases over the layers, i.e.

Neural networks meeting this criteria have a number of weight parameters that is larger than the cardinality of the state space, making them impractical for large |S| . While ongoing work seeks to relax or remove this assumption, we point out that similar over-parameterization assumptions are critical and very common in most existing works on the theory of neural networks [Zou and Gu, 2019, Nguyen, 2019, Liu et al., 2022, Martinetz and Martinetz, 2022, Pandey and Kumar, 2023].

The λ -superlevel set of the value function with respect to θ under reward function r is

Our next main theoretical result guarantees the connectedness of U Ω λ,r .

Theorem 2 Under Assumptions 1-3, the superlevel set U Ω λ,r is connected for any λ ∈ R . In addition, with J r, Ω ( θ ) ≜ J r ( π θ ) , the collection of functions { J r, Ω ( · ) : Ω → R } r ∈ R |S|×|A| is equiconnected.

The proof of this theorem is deferred to the appendix. Similar to Theorem 1, the claim in Theorem 2 on the equiconnectedness of { J r, Ω } r ∈ R |S|×|A| is again stronger than the connectedness of U Ω λ,r and needs to be derived for the application to minimax theorems, which we discuss in the next section.

4 Application to Robust Reinforcement Learning

In this section, we consider the robust RL problem under adversarial reward attack, which can be formulated as a convex-nonconcave minimax optimization program. In Section 4.1, we show that the minimax equality holds for this optimization program in the tabular policy setting and under policies represented by a class of neural networks, as a consequence of our results in Sections 2 and 3. To our best knowledge, the existence of the Nash equilibrium for this robust RL problem has not been established before even in the tabular case. A specific example of this type of robust RL problems is given in Section 4.2.

4.1 Minimax Theorem

Robust RL in general studies identifying a policy with reliable performance under uncertainty or attacks. A wide range of formulations have been proposed for robust RL (which we reviewed in details in Section 1.2), and an important class of formulations takes the form of defending against an adversary that can modify the reward function in a convex manner. Specifically, the objective of the learning agent can be described as solving the following minimax optimization problem

where C is some convex set. It is unclear from the existing literature whether minimax inequality holds for this problem, i.e.

and we provide a definitive answer to this question. We note that there exists a classic minimax theorem on a special class of convex-nonconcave functions [Simons, 1995], which we adapt and simplify as follows.

Theorem 3 Consider a separately continuous function f : X × Y → R , with Y being a convex, compact set. Suppose that f ( x, · ) is convex for all x ∈ X . Also suppose that the collection of functions { f ( · , y ) } y ∈Y is equiconnected. Then, we have

Theorem 3 states that the minimax equality holds under two main conditions (besides the continuity condition, which can easily be verified to hold for J r ( π ) ). First, the function f ( x, y ) needs to be convex with respect to the variable y within a convex, compact constraint set. Second, f ( x, y ) needs to have a connected superlevel set with respect to x , and the path function constructed to prove the connectedness of the superlevel set is independent of y . As we have shown in this section and earlier in the paper, if we model J r ( π ) by f ( x, y ) with π and r corresponding to x and y , both conditions are observed by the optimization problem (5), which allows us to state the following corollary.

Corollary 1 Suppose that the Markov chain M satisfies Assumption 1 on ergodicity. Then, the minimax equality (6) holds.

When the neural network presented in Section 3 is used to represent the policy, the collection of functions { J r, Ω } r is also equiconnected. This allows us to extend the minimax equality above to the neural network policy class. Specifically, consider problem (5) where the policy π θ is represented by the parameter θ ∈ Ω as described in Section 3. Using f ( x, y ) to model J r ( π θ ) with x and y mirroring θ and r , we can easily establish the minimax theorem in this case as a consequence of Theorem 2 and 3.

Corollary 2 Suppose that the Markov chain M satisfies Assumption 1 on ergodicity and that the neural policy class satisfies Assumptions 2-3. Then, we have

Corollary 1 and 2 establish the minimax equality (or equivalently, the existence of the Nash equilibrium) for the robust reinforcement learning problem under adversarial reward attack for the tabular and neural network policy class, respectively. To our best knowledge, these results are both novel and previously unknown in the existing literature. The Nash equilibrium is an important global optimality notion in minimax optimization, and the knowledge on its existence can provide strong guidance on designing and analyzing algorithms for solving the problem.

4.2 Example - Defense Against Reward Poisoning

We now discuss a particular example of (5). We consider the infinite horizon, average reward MDP M = ( S , A , P , r ) introduced in Section 2, where r is the true, unpoisoned reward function. Let Π det denote the set of deterministic policies from S to A . With the perfect knowledge of this MDP, an attacker has a target policy π † ∈ Π det and tries to make the learning agent adopt the policy by

manipulating the reward function. Mathematically, the goal of the attacker can be described by the function Attack( r, π † , ϵ † ) which returns a poisoned reward under the true reward r , the target policy π † , and a pre-selected margin parameter ϵ † ≥ 0 . Attack( r, π † , ϵ † ) is the solution to the following optimization problem

In other words, the attacker needs to minimally modify the reward function to make π † the optimal policy under the poisoned reward. This optimization program minimizes a quadratic loss under a finite number of linear constraints and is obviously convex.

The learning agent observes the poisoned reward r † = Attack( r, π † , ϵ † ) rather than the original reward r . As noted in Banihashem et al. [2021], without any defense, the learning agent solves the policy optimization problem under r † to find π † , which may perform arbitrarily badly under the original reward. One way to defend against the attack is to maximize the performance of the agent in the worst possible case of the original reward, which leads to solving a minimax optimization program of the form

When the policy π is fixed, (10) reduces to

With the justification deferred to Appendix D, we point out that (11) consists of a linear objective function and a convex (and compact) constraint set, and is therefore a convex program. On the other side, when we fix the reward r ′ , (10) reduces to a standard policy optimization problem.

We are interested in investigating whether the following minimax equality holds.

This is a special case of (5) with C = { r ′ | Attack( r ′ , π † , ϵ † ) = r † } , which can be verified to be a convex set. Therefore, the validity of (12) directly follows from Corollary 1. Similarly, in the setting of neural network parameterized policy we can establish

as a result of Corollary 2.

5 Conclusions & Future Work

We study the superlevel set of the policy optimization problem under the MDP framework and show that it is always a connected set under a tabular policy and for policies parameterized by a class of neural networks. We apply this result to derive a previously unknown minimax theorem for a robust RL problem. An immediate future direction of the work is to investigate whether/how the result discussed in this paper can be used to design better RL algorithms. In Fatkhullin and Polyak [2021], the authors show that the original LQR problem has connected level sets, but the partially observable LQR does not. It is interesting to study whether this observation extends to the MDP setting, i.e. the policy optimization problem under a partially observable MDP can be shown to have disconnected superlevel sets.

References

Leonard Adolphs, Hadi Daneshmand, Aurelien Lucchi, and Thomas Hofmann. Local saddle point optimization: A curvature exploitation approach. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 486-495. PMLR, 2019.

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference on Learning Theory , pages 64-66. PMLR, 2020.

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research , 22(1):4431-4506, 2021.

Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine , 34(6):26-38, 2017.

Kiarash Banihashem, Adish Singla, and Goran Radanovic. Defense against reward poisoning attacks in reinforcement learning. arXiv preprint arXiv:2102.05776 , 2021.

Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXiv preprint arXiv:1906.01786 , 2019.

Jalaj Bhandari and Daniel Russo. On the linear convergence of policy gradient methods for finite mdps. In International Conference on Artificial Intelligence and Statistics , pages 2386-2394. PMLR, 2021.

Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 70(4): 2563-2578, 2022.

Constantinos Daskalakis and Ioannis Panageas. The limit points of (optimistic) gradient descent in min-max optimization. Advances in neural information processing systems , 31, 2018.

J-Cl Evard and Farhad Jafari. The set of all m × n rectangular real matrices of rank r is connected by analytic regular arcs. Proceedings of the American Mathematical Society , 120(2):413-419, 1994.

Ky Fan. Minimax theorems. Proceedings of the National Academy of Sciences , 39(1):42-47, 1953.

Ilyas Fatkhullin and Boris Polyak. Optimizing static linear feedback: Gradient method. SIAM Journal on Control and Optimization , 59(5):3887-3911, 2021.

Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International conference on machine learning , pages 1467-1476. PMLR, 2018.

Robert Gower, Othmane Sebbouh, and Nicolas Loizou. Sgd for structured nonconvex functions: Learning rates, minibatching and interpolation. In International Conference on Artificial Intelligence and Statistics , pages 1315-1323. PMLR, 2021.

Vineet Goyal and Julien Grand-Clement. Robust markov decision processes: Beyond rectangularity. Mathematics of Operations Research , 2022.

Aaron J Havens, Zhanhong Jiang, and Soumik Sarkar. Online robust policy learning in the presence of unknown adversaries. arXiv preprint arXiv:1807.06064 , 2018.

Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality in nonconvex-nonconcave minimax optimization? In International conference on machine learning , pages 4880-4889. PMLR, 2020.

Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximalgradient methods under the Polyak-łojasiewicz condition. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I 16 , pages 795-811. Springer, 2016.

Vijay R. Konda. Actor-critic algorithms . PhD thesis, Massachusetts Institute of Technology, 2002.

Yan Li, Tuo Zhao, and Guanghui Lan. First-order policy optimization for robust markov decision process. arXiv preprint arXiv:2209.10579 , 2022.

Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis , 59: 85-116, 2022.

Julius Martinetz and Thomas Martinetz. Highly over-parameterized classifiers generalize since bad solutions are rare. arXiv preprint arXiv:2211.03570 , 2022.

Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning , pages 6820-6829. PMLR, 2020.

Hesameddin Mohammadi, Armin Zare, Mahdi Soltanolkotabi, and Mihailo R Jovanovi´ c. Convergence and sample complexity of gradient methods for the model-free linear-quadratic regulator problem. IEEE Transactions on Automatic Control , 67(5):2435-2450, 2021.

J v Neumann. Zur theorie der gesellschaftsspiele. Mathematische annalen , 100(1):295-320, 1928.

Quynh Nguyen. On connected sublevel sets in deep learning. In International Conference on Machine Learning , pages 4790-4799. PMLR, 2019.

Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. Solving a class of non-convex min-max games using iterative first order methods. Advances in Neural Information Processing Systems , 32, 2019.

Kishan Panaganti and Dileep Kalathil. Sample complexity of robust reinforcement learning with a generative model. In International Conference on Artificial Intelligence and Statistics , pages 9582-9602. PMLR, 2022.

Eshan Pandey and Santosh Kumar. Exploring the generalization capacity of over-parameterized networks. International Journal of Intelligent Systems and Applications in Engineering , 11(1s): 97-112, 2023.

Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning , pages 2817-2826. PMLR, 2017.

Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning. In International Conference on Machine Learning , pages 7974-7984. PMLR, 2020.

Anindya Sarkar, Jiarui Feng, Yevgeniy Vorobeychik, Christopher Gill, and Ning Zhang. Reward delay attacks on deep reinforcement learning. arXiv preprint arXiv:2209.03540 , 2022.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Stephen Simons. Minimax theorems and their proofs. In Minimax and applications , pages 1-23. Springer, 1995.

Maurice Sion. On general minimax theorems. Pacific Journal of mathematics , 8(1):171-176, 1958.

Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning , pages 6215-6224. PMLR, 2019.

Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008.01825 , 2020.

Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 6202-6209, 2020.

Mengdi Wang. Primal-dual π learning: Sample complexity and sublinear run time for ergodic markov decision problems. arXiv preprint arXiv:1710.06100 , 2017.

Yue Wang, Alvaro Velasquez, George Atia, Ashley Prater-Bennette, and Shaofeng Zou. Robust average-reward markov decision processes. arXiv preprint arXiv:2301.00858 , 2023.

Chen-Yu Wei, Mehdi Jafarnia Jahromi, Haipeng Luo, Hiteshi Sharma, and Rahul Jain. Modelfree reinforcement learning in infinite-horizon average-reward markov decision processes. In International conference on machine learning , pages 10170-10180. PMLR, 2020.

Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems , 33:17617-17628, 2020.

Lin Xiao. On the convergence rates of policy gradient methods. The Journal of Machine Learning Research , 23(1):12887-12922, 2022.

Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 , 2015.

Junchi Yang, Negar Kiyavash, and Niao He. Global convergence and variance reduction for a class of nonconvex-nonconcave minimax problems. Advances in Neural Information Processing Systems , 33:1153-1165, 2020.

Zhuoran Yang, Yongxin Chen, Mingyi Hong, and Zhaoran Wang. Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost. Advances in neural information processing systems , 32, 2019.

Sihan Zeng, Malik Aqeel Anwar, Thinh T Doan, Arijit Raychowdhury, and Justin Romberg. A decentralized policy gradient approach to multi-task reinforcement learning. In Uncertainty in Artificial Intelligence , pages 1002-1012. PMLR, 2021a.

Sihan Zeng, Thinh T Doan, and Justin Romberg. A two-time-scale stochastic optimization framework with applications in control and reinforcement learning. arXiv preprint arXiv:2109.14756 , 2021b.

Sihan Zeng, Thinh Doan, and Justin Romberg. Regularized gradient descent ascent for two-player zero-sum markov games. Advances in Neural Information Processing Systems , 35:34546-34558, 2022.

Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems , 33:21024-21037, 2020.

Difan Zou and Quanquan Gu. An improved analysis of training over-parameterized deep neural networks. Advances in neural information processing systems , 32, 2019.

A Proof of Theorems

A.1 Proof of Theorem 1:

We note that there exists a bijective map between π and ̂ µ π where ̂ µ π is induced by π according to (2) and conversely

provided that µ π ( s ) = 0 , which is guaranteed by Assumption 1. Eq. (13) inspires the construction of the path map.

̸

To prove that the superlevel set is connected, we show that for any λ ∈ R and π 1 , π 2 ∈ U λ,r , there exists a continuous path map p : [0 , 1] →U λ,r such that p (0) = π 1 and p (1) = π 2 . We now construct the path function p by defining

which is well-defined for all α ∈ [0 , 1] as µ π 1 ( s ) , µ π 2 ( s ) are positive for all s ∈ S . Note that the construction of p does not depend on the reward function r . It is easy to see that p ( α ) ∈ ∆ S A is a continuous in α . To stress that p ( α ) is in the policy space, we denote π α = p ( α ) .

Recall the definition of the transition probability matrix in (1). We define B ∈ R |S| as

Each entry of B can be expressed as

which implies

Aconsequence of Assumption 1 is that for any policy π there is a unique eigenvector of P π associated with the eigenvalue 1 , and this eigenvector (properly normalized) is the stationary distribution. Therefore, (14) means that αµ π 1 +(1 -α ) µ π 2 has to be the stationary distribution under policy π α , i.e.

As a result, for all s ∈ S , a ∈ A

Note that J r ( π ) = ∑ s ∈S ,a ∈A r ( s, a ) ̂ µ π ( s, a ) . Since π π 1 , π π 2 ∈ U λ,r , we know

Therefore, we have for any α ∈ [0 , 1]

which implies π α ∈ U λ,r . So far we have verifed that the constructed path map p is indeed continuous and maps α ∈ [0 , 1] to U λ,r with p (0) = π 1 and p (1) = π 2 . This concludes the proof on the connectedness of the superlevel set U λ,r . The claim on the equiconnectedness simply follows from the fact that the construction of the path map p does not depend on the reward function.

■

A.2 Proof of Theorem 2

We use X to denote the concatenation of the feature vectors across all states

In the analysis we may apply the softmax function ψ to a matrix in a row-wise fashion. Specifically, for any n ≥ 1 and matrix M ∈ R n ×|A| , we have

The softmax operator ψ can be inverted up to an additive constant factor. We define ψ inv for any matrix M ∈ R n ×|A| as

with c i determined such that ∑ |A| j =1 ψ inv ( M ) i,j = 0 . Note that ψ inv is a right inverse of ψ , i.e. ψ ( ψ inv ( M )) = M for all matrix A .

When the input to a neural network with parameter θ is the feature table X , we denote the output of layer k by F θ k ∈ R |S|× n k . According to (4), F θ k can be expressed as

where 1 |S| is the all-one vector of dimension |S| × 1 . Note that F θ L ∈ R |S|×|A| is the policy table produced by the neural network, i.e. π θ = F θ L .

The proof of Theorem 2 relies on the following intermediate results, which we now present. The proof of Proposition 1 can be found in Appendix B.

Proposition 1 If rank( X ) = |S| , then under Assumption 1 and 2, the superlevel set U Ω λ,r is connected for all λ ∈ R .

Lemma 1 Let ( X,W,b,V ) ∈ R |S|× n 0 × R n 0 × n 1 × R n 1 × R n 1 × n 2 . Let Z = σ ( XW +1 |S| b ⊤ ) V . Suppose X has distinct rows. Then, under Assumption 2 and 3, there exists a continuous path map c : [0 , 1] → R n 0 × n 1 × R n 1 × R n 1 × n 2 with c ( λ ) = ( W ( λ ) , b ( λ ) , V ( λ )) such that

1) c(0) = (W,b,V),

Lemma 2 Let ( X,W,V,W ′ ) ∈ R |S|× n 0 × R n 0 × n 1 × R n 1 × n 2 × R n 0 × n 1 . Suppose rank( σ ( XW )) = |S| and rank( σ ( XW ′ )) = |S| . Then, under Assumption 2 and 3, there exists a continuous path map c : [0 , 1] → R n 0 × n 1 × R n 1 × n 2 with c ( λ ) = ( W ( λ ) , V ( λ )) such that

To prove Theorem 2, it suffices to show that for any θ 1 = ( W 1 ,l , b 1 ,l ) L l =1 ∈ U Ω λ,r and θ 2 = ( W 2 ,l , b 2 ,l ) L l =1 ∈ U Ω λ,r there exists a connected path that is completely within U Ω λ,r .

Applying Lemma 1 with ( X,W 1 , 1 , b 1 , 1 , W 1 , 2 ) and ( X,W 2 , 1 , b 2 , 1 , W 2 , 2 ) , the problem simplifies to showing the existence of a continuous path within U Ω λ,r that connects

and such that

Then, we can apply Lemma 2 with ([ X, 1 |S| ] , [ W ′⊤ 1 , 1 , b ′ 1 , 1 ] ⊤ , W ′ 1 , 2 , [ W ′⊤ 2 , 1 , b ′ 2 , 1 ] ⊤ ) to show that there is a continuous path between θ ′ 1 and θ ′′ 1 with θ ′′ 1 = (( W ′ 2 , 1 , b ′ 2 , 1 ) , ( W ′′ 1 , 2 , b 1 , 2 ) , ( W 1 ,l , b 1 ,l ) L l =3 ) such that

As a consequence, now we simply have to show that θ ′′ 1 and θ ′ 2 is connected by a continuous path within U Ω λ,r .

Note that θ ′′ 1 and θ ′ 2 have identical first layer parameters and thus the same first layer output, which is full rank. This allows us to treat the layers from 2 to L as a new network and apply Proposition 1 (which requires the input to be full rank) to the new network to guarantee that there exists a continuous path map c : [0 , 1] → Ω 2 × ... × Ω k such that c (0) = (( W ′′ 1 , 2 , b 1 , 2 ) , ( W 1 ,l , b 1 ,l ) L l =3 ) ,

for all α ∈ [0 , 1] . This implies that there is indeed a continuous path between θ ′′ and θ ′ within U Ω .

1 2 λ,r

Similar to the proof of Theorem 1, the claim on the connectedness simply follows from the fact that the construction of the path map p does not depend on the reward function. ■

B Proof of Proposition 1

For each layer of the neural network k = 1 , · · · , L , we define Ω ⋆ k ⊆ Ω k to be the set of weights W k and biases b k of layer k such that W k is full rank, i.e.

We denote Ω ⋆ = Ω ⋆ 1 × Ω ⋆ 2 × ... × Ω ⋆ L . Next, we introduce the following lemmas in aid of the analysis.

Condition 1 Given θ = ( W l , b l ) L l =2 , W l has full rank for every l ∈ [2 , L ] .

Lemma 3 Under Assumption 2, 3, and Condition 1, given any k ∈ [2 , L ] and matrix F ∈ R |S|× n k , there exists a continuous map h : Ω ⋆ 2 × ... × Ω ⋆ k × R |S|× n k → Ω 1 such that

1) Given (( W 2 , b 2 ) , ..., ( W k , b k ) , F ) ∈ Ω ⋆ 2 × ... × Ω ⋆ k × R |S|× n k , we have

2) For any θ ⋆ = ( W ⋆ l , b ⋆ l ) L l =1 ∈ Ω 1 × Ω ⋆ 2 × ... × Ω ⋆ L , there exists a continuous path map p : [0 , 1] → Ω 1 × Ω ⋆ 2 × ... × Ω ⋆ L such that p (0) = θ ⋆ , p (1) = ( h (( W ⋆ l , b ⋆ l ) k l =2 , F θ ⋆ k ) , ( W ⋆ l , b ⋆ l ) L l =2 ) , and F p ( α ) L = F θ ⋆ L for all α ∈ [0 , 1] .

Lemma 4 Given two connected sets A ⊆ R m 1 × n and B ⊆ R n × m 2 , the set { ab | a ∈ A , b ∈ B} is connected. Given two connected sets A , B ⊆ R m × n , the set { a + b | a ∈ A , b ∈ B} is connected.

Lemma 5 Under Assumption 2, for any θ ∈ Ω , there exist θ ⋆ ∈ Ω ⋆ and a continuous path map p : [0 , 1] → Ω such that p (0) = θ , p (1) = θ ⋆ , and F p ( α ) L = F θ L for all α ∈ [0 , 1] .

Lemma 6 If n < m , then the set F = { F ∈ R m × n | rank( F ) = n } is connected. In other words, given F 1 , F 2 ∈ F , there exists a continuous path map q : [0 , 1] → F such that q (0) = F 1 and q (1) = F 2 .

Fix a λ ∈ R . To show the superlevel set U Ω λ,r is connected, it suffices to show that for any θ 1 , θ 2 ∈ U Ω λ,r , there exists a continuous path between them that is completely in U Ω λ,r .

Without any loss of generality, we can safely assume that both θ 1 = ( W 1 ,l , b 1 ,l ) L l =1 and θ 2 = ( W 2 ,l , b 2 ,l ) L l =1 satisfy Condition 1, since otherwise by Lemma 5 we can find a continuous path from θ 1 and θ 2 that leads to one satisfying Condition 1. We denote the policies parameterized by θ 1 , θ 2 as π 1 , π 2 , i.e.

By Lemma 3, there is a continuous path from θ 1 /θ 2 to θ ′ 1 /θ ′ 2 where we define

Now, we just have to show that there exists a continuous path between θ ′ 1 and θ ′ 2 that is completely within U Ω λ,r . By Lemma 6, we know that for l = 2 , .., L , there exists a continuous path map q l : [0 , 1] → Ω ⋆ l such that q l (1) = W 1 ,l and q l (0) = W 2 ,l . Then, we construct the map q : [0 , 1] → Ω

It is obvious that q is a continuous map as h, q 2 , ..., q L are continuous. In addition, F q ( α ) L = π 1 for all α ∈ [0 , 1] , and q (1) = θ ′ 1 . We define

Now our aim simplifies to finding a continuous path between θ ′′ 1 and θ ′ 2 that is completely in U Ω λ,r . To show that this path exists, we construct a continuous map t : [0 , 1] → Ω as follows

where ˜ π is defined entry-wise

It can be seen that t is indeed continuous since ˜ π α is continuous in α , and t (0) = θ ′ 2 and t (1) = θ ′′ 1 . What remains to be shown is that F t ( α ) L ∈ U Ω λ,r , i.e. J r ( F t ( α ) L ) ≥ λ . By the definition of h in Lemma 3, F t ( α ) L = ˜ π α . It has been shown in the proof of Theorem 1 that indeed J r (˜ π α ) ≥ λ provided that J r ( π 1 ) ≥ λ and J r ( π 2 ) ≥ λ . This concludes the proof of Proposition 1.

■

C Proof of Supporting Lemmas

C.1 Proof of Lemma 1

This lemma is adapted from Lemma 5.2 of Nguyen [2019].

C.2 Proof of Lemma 2

This lemma is adapted from Lemma 5.3 of Nguyen [2019].

C.3 Proof of Lemma 3

We provide a proof for the case k = L . For k = L , the proof can be found in Nguyen [2019][Lemma 3.3].

̸

For (( W 2 , b 2 ) , ..., ( W L , b L ) , π ) ∈ Ω ⋆ 2 × ... × Ω ⋆ L × ∆ S A , we define the map h as follows

where ̂ W 1 and ̂ b 1 is defined as

where we use A † to denote the Moore-Penrose inverse of a matrix A . If A has full column rank, then we have A † A = I . If A has full row rank, we have AA † = I . We can easily see that the defined h operator is continuous as it is a composition of continuous operators.

Assumption 3, and Condition 1 imply that the matrices W 2 , ..., W L all have full column rank, which means W † l W l = I . We also know that [ X, 1 |S| ] has full row rank by our assumption that X has full row rank, which means [ X, 1 |S| ][ X, 1 |S| ] † = I . Therefore, we can layerwise invert (16) and verify that

For every layer l = 2 , ..., L , we define the operator G l : R |S|× n l -1 → R |S|× n l

We also define the operator H : R ( d +1) × n 1 → R |S|× n 1

To show the continuous path claimed in Lemma 3 exists, it suffices to show that the set { ( W 1 , b 1 ) : F ( W 1 ,b 1 ) , ( W ⋆ l ,b ⋆ l ) L l =2 L = F θ ⋆ L } is connected, which is equivalent to showing that the set f -1 ( F θ ⋆ L ) is connected where f is defined as

Note that the definition of f implies

̸

Note that G -1

̸

It is easy to verify that { C | C i,j = C i,j ′ ∀ i, j = j ′ } and { B | BW ⋆ l = 0 } for all l = 2 , ..., L . Then, Lemma 4 implies that G -1 l ( F ) is a connected set for all F .

Similarly, H -1 ( F ) = [ X, 1 |S| ] † σ -1 ( F ) + { B | [ X, 1 |S| ] B = 0 } is also a connected set for all F . Therefore, from (17) we know that f -1 ( F ) is a connected set for any F , which concludes the proof of Lemma 3.

■

C.4 Proof of Lemma 4

To show that the product of the two connected sets are connected, we consider any x, y ∈ { ab | a ∈ A , b ∈ B} . Obviously, there exist a x , a y ∈ A and b x , b y ∈ B such that x = a x b x , y = a y b y . The connectedness of A and B implies that there exists continuous path maps p A : [0 , 1] →A and p B : [0 , 1] →B such that p A (0) = a x , p A (1) = a y , p B (0) = b x , p B (1) = b y . Define p ( α ) = p A p B for α ∈ [0 , 1] . It is obvious that p ( α ) ∈ { ab | a ∈ A , b ∈ B} for all α . Since the product of continuous maps is still continuous, p : [0 , 1] → { ab | a ∈ A , b ∈ B} is a continuous path map satisfying p (0) = x and p (1) = y . This implies that the set { ab | a ∈ A , b ∈ B} is a connected set.

A similar argument can be used to show that the sum of two connected sets is connected.

■

C.5 Proof of Lemma 5

Define ˜ F L (( W l , b l ) L l =1 ) as the output of the final layer before the softmax activation

Then, existing results in the literature (such as Lemma 3.4 of Nguyen [2019]) show that for any θ ∈ Ω , there exists a continuous path map p : [0 , 1] → Ω such that p (0) = θ , p (1) = θ ⋆ ∈ Ω ⋆ , and ˜ F L ( p ( α )) = ˜ F L ( θ ) for all α ∈ [0 , 1] . This leads to our claim.

■

C.6 Proof of Lemma 6

This lemma is adapted from Theorem 4 of Evard and Jafari [1994].

D Convexity of Optimization Program (11)

In this section, we show that (11) is a convex optimization program. First, we note that

which means that the objective function is linear in the reward.

The constraint set is obvious closed. It is also bounded as the reward r ( s, a ) ∈ [0 , U r ] . To prove the constraint set is convex, we need to show that for any r 1 , r 2 such that Attack( r 1 , π † , ϵ † ) = r † and Attack( r 2 , π † , ϵ † ) = r † , we have

By the optimality condition of (9), r † being the optimal poisoned reward for true reward r 1 and r 2 is equivalent to

for all r such that J r ( π † ) ≥ J r ( π ) + ϵ † , ∀ π ∈ Π det \ π † . By taking the convex combination of these two inequalities, we have for any α ∈ [0 , 1]

for all r such that J r ( π † ) ≥ J r ( π ) + ϵ † , ∀ π ∈ Π det \ π † . Again by the optimality condition of (9), (19) is equivalent to (18).

At this point, we have shown that (11) has a linear objective function and a convex (and also compact) constraint set. As a result, the optimization program is convex.

Peer reviews:
comments: SUMMARY:
Authors found novel discoveries in policy optimization problems: 1.the superlevel set of the objective function related to the policy parameter is always a connected set and the optimization objective as a function of the policy parameter and reward satisfies a stronger “equiconnectedness” property. Based on the discoveries, authors derive a novel minimax theorem for a robust RL problem.

STRENGTHS:
Well written and well organized.
Novel discoveries of policy optimization problems in reinforcement learning. 
Derived a novel minimax theorem for a robust RL problem which may contribute to the development of novel robust algorithm in the future. Theoretical analysis and proofs are nicely presented.

WEAKNESSES:
Checking math is not my strength so I'll refrain from providing my opinion on the math. I think one weakness is about the application side of the paper and lack of any experiments(maybe just a toy example?).

QUESTIONS:
Would it be possible to design a toy env and run some small experiments to explicitly show how the derived theorem can help develop better algorithm in addition to the reward poisoning example?

LIMITATIONS:
Again, checking math is not my strength so I'll refrain from providing my opinion on the math. One limitation is lack of experiments and application.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This work shows the superlevel set of the objective function in reinforcement learning is always equiconnected for both tabular policy and neural policy. An application of the connected property is the minimax theorem. As a consequence, reward attack robust RL can be shown to have Nash equilibrium.

STRENGTHS:
1. The paper show the superlevel set of the RL objective is connected. Furthermore, the collection of the objective functions under the tabular and the neural policy is equiconnected. 
2. The connected property is used to establish the minimax theorem.
3. As a corollary, the authors show that reward attack robust RL problems have Nash equilibrium. This is the first work that shows such a result. 
4. The paper is well-written.

WEAKNESSES:
1. Although Theorem 1 and 2 are general results that cover a large class of policies, it is not clear whether the minimax theorem can be used to show the existence of Nash equilibrium for other robust RL problems, e.g., when the agent is uncertain of the transition kernels. 
2. It is unclear whether the theoretical results can help design algorithms for robust RL problems.

QUESTIONS:
I have some minor questions:
1. What is the intuition behind Assumption 2? Why do we need the non-zero scalar condition?
2. Can we use the equiconnected property to prove minimax theorems other than Theorem 3?

LIMITATIONS:
The authors address limitations in the conclusion section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper aims to enhance the comprehension of the optimization landscape in reinforcement learning (RL) for policy optimization problems. The primary contribution of this work is to demonstrate the connectedness of the superlevel set of the policy optimization problem in RL under a tabular policy representation. Furthermore, the authors establish that the superlevel set of the objective function, considering the policy parameters (i.e., weights of the neural networks), remains connected across all levels. The authors also illustrate the practical implications of their main findings by deriving a minimax theorem for a specific class of robust RL problems.

STRENGTHS:
- The paper studies an exciting area of research on optimization applied to policy learning in reinforcement learning. 
- The paper is generally well written and coherent.

WEAKNESSES:
- The paper briefly talks about robust RL as a potential application of studying super level connectedness. It does not provide any practical robust RL algorithm.

QUESTIONS:
- I would like the authors to include an explanation for why one needs to expand their understanding of RL optimization landscape beyond gradient domination. To motivate the problem to the general conference audience, it is important to list additional insights one might gain from studying super level set connectedness (SLSC). For example, does SLSC help identifying bottlenecks, regions of poor convergence, or potential areas for algorithmic improvement? 
- It is also not clear to me what were the exact challenges in establishing results for deep RL? 
- Are the results in Sec 2 similar to Nguyen [2019]? If not, what are the exact differences?

LIMITATIONS:
The authors are clear on the limitations of the work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the connectedness in (deep) reinforcement learning. First, the authors show that the superlevel set of average reward objective in reinforcement learning is connected under both tabular and over-parameterized policies. The objective is shown to satisfy a stronger equiconnectedness property. Second, the authors use the results to get minimax theorems for robust reinforcement learning. In particular, they show that show that minimax problems with convex functions on one side and equiconnected functions on the other side observes the minimax equality (i.e. has a Nash equilibrium).\n\nSTRENGTHS:\nThe superlevel set of average reward objective in reinforcement learning is connected seems a novel and interesting result.\n\nThe results holds for over-parameterized neural networks, and find applications in minimax problems.\n\nWEAKNESSES:\nSome simulations could be used to verify the results.\n\nQUESTIONS:\n1. The authors showed that gradient dominance and connectedness do not imply one another. This raises the question of whether connectedness properties of objective functions allow any algorithms to find optimal solutions?\n\n2. Related question: It seems that in Figure 1, the left example is easy for optimization (separated global maximizers), while the right example seems to be harder for optimization (stationary points which are not global maximizer). Does this imply that comparing to gradient dominance, connectedness is a less favourable property for optimization?\n\n3. There are existing results showing that in game settings, a property called pseudo-linearity is satisfied, i.e., there are monotonically increasing path between two policies, see Theorem 1 of https://arxiv.org/abs/2204.04186. I am curious if there is any relation between the connectedness in this work and pseudo-linearity in the above paper?\n\nLIMITATIONS:\nThis is a mostly theoretical work. There is no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-05"}