SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sihan Zeng
Atlanta, GA 30318
szeng30@gatech.edu
Thinh T. Doan
Blacksburg, VA 24061
thinhdoan@vt.edu
Justin Romberg
jrom@ece.gatech.edu
The aim of this paper is to improve the understanding of the optimization landscape for policy optimization problems in reinforcement learning. Specifically, we show that the superlevel set of the objective function with respect to the policy parameter is always a connected set both in the tabular setting and under policies represented by a class of neural networks. In addition, we show that the optimization objective as a function of the policy parameter and reward satisfies a stronger "equiconnectedness" property. To our best knowledge, these are novel and previously unknown discoveries.
We present an application of the connectedness of these superlevel sets to the derivation of minimax theorems for robust reinforcement learning. We show that any minimax optimization program which is convex on one side and is equiconnected on the other side observes the minimax equality (i.e. has a Nash equilibrium). We find that this exact structure is exhibited by an interesting class of robust reinforcement learning problems under an adversarial reward attack, and the validity of its minimax equality immediately follows. This is the first time such a result is established in the literature.
Policy optimization problems in reinforcement learning (RL) are usually formulated as the maximization of a non-concave objective function over a convex constraint set. Such non-convex programs are generally difficult to solve globally, as gradient-based optimization algorithms can be trapped in sub-optimal first-order stationary points. Interestingly, recent advances in RL theory [Fazel et al., 2018, Agarwal et al., 2021, Mei et al., 2020] have discovered a "gradient domination" structure in the optimization landscape, which qualitatively means that every stationary point of the objective function is globally optimal. An important consequence of this condition is that any first-order algorithm that converges to a stationary point is guaranteed to find the global optimality.
In this work, our aim is to enhance the understanding of the optimization landscape in RL beyond the gradient domination condition. Inspired by Mohammadi et al. [2021], Fatkhullin and Polyak [2021] that discuss properties of the sublevel set for the linear-quadratic regulator (LQR), we study
the superlevel set of the policy optimization objective under a Markov decision process (MDP) framework and prove that it is always connected.
As an immediate consequence, we show that any minimax optimization program which is convex on one side and is an RL objective on the other side observes the minimax equality. We apply this result to derive an interesting and previously unknown minimax theorem for robust RL. We also note that it is unclear at the moment, but certainly possible, that the result on connected superlevel sets may be exploited to design more efficient and reliable policy optimization algorithms in the future.
1.1 Main Contribution
Our first contribution in this work is to show that the superlevel set of the policy optimization problem in RL is always connected under a tabular policy representation. We then extend this result to the deep reinforcement learning setting, where the policy is represented by a class of over-parameterized neural networks. We show that the superlevel set of the underlying objective function with respect to the policy parameters (i.e. weights of the neural networks) is connected at all levels. We further prove that the policy optimization objective as a function of the policy parameter and reward is "equiconnected", which is a stronger result that we will define and introduce later in the paper. To the best of our knowledge, our paper is the first to rigorously investigate the connectedness of the superlevel sets for the MDP policy optimization program, both in the tabular case and with a neural network policy class.
As a downstream application, we discuss how our main results can be used to derive a minimax theorem for a class of robust RL problems. We consider the scenario where an adversary strategically modifies the reward function to trick the learning agent. Aware of the attack, the learning agent defends against the poisoned reward by solving a minimax optimization program. The formulation for this problem is proposed and considered in Banihashem et al. [2021], Rakhsha et al. [2020]. However, as a fundamental question, the validity of the minimax theorem (or equivalently, the existence of a Nash equilibrium) is still unknown. We fill in this gap by establishing the minimax theorem as a simple consequence of the equiconnectedness of the policy optimization objective.
Our paper is closely connected to the existing works that study the structure of policy optimization problems in RL, especially those on the gradient domination condition. Our result also relates to the literature on minimax optimization for various function classes and robust RL. We discuss the recent advances in these domains to give context to our contributions.
Gradient Domination Condition. The policy optimization problem in RL is non-convex but obeys the special "gradient domination" structure, which has been widely used as a tool to show the convergence of various gradient-based algorithms to the globally optimal policy [Agarwal et al., 2020, Mei et al., 2020, Bhandari and Russo, 2021, Zeng et al., 2021a, Xiao, 2022]. In the settings of LQR [Fazel et al., 2018, Yang et al., 2019] and entropy-regularized MDP [Mei et al., 2020, Cen et al., 2022, Zeng et al., 2022], the gradient domination structure can be mathematically described by the Polyak-Lojasiewicz (PL) condition, which bears a resemblance to strong convexity but does not even imply convexity. It is known that functions observing this condition can be optimized globally and efficiently by (stochastic) optimization algorithms [Karimi et al., 2016, Zeng et al., 2021b, Gower et al., 2021]. When the policy optimization problem under a standard, non-regularized MDP is considered, the gradient domination structure is weaker than the PL condition but still takes the form of upper bounding a global optimality gap by a measure of the magnitude of the gradient [Bhandari and Russo, 2019, Agarwal et al., 2020, 2021]. In all scenarios, the gradient domination structure prevents any stationary point from being sub-optimal.
It may be tempting to think that the gradient domination condition and the connectedness of the superlevel sets are strongly connected notions or may even imply one another. For 1-dimensional function  $(f:\mathbb{R}^n\to \mathbb{R}$  with  $n = 1$ ), it is easy to verify that the gradient domination condition necessarily implies the connectedness of the superlevel sets. However, when  $n\geq 2$  this is no longer true. In general, the gradient domination condition neither implies nor is implied by the connectedness of superlevel sets, which we illustrate with examples in Section 1.3. These two structural properties are distinct concepts that characterize the optimization landscape from different angles. This observation precludes the possibility of deriving the connectedness of the superlevel
sets in RL simply from the existing results on the gradient domination condition, and suggests that a tailored analysis is required.
Minimax Optimization & Minimax Theorems. Consider a function  $f: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$  on convex sets  $\mathcal{X}, \mathcal{Y}$ . In general, the minimax inequality always holds
$$
\sup_{x\in \mathcal{X}}\inf_{y\in \mathcal{Y}}f(x,y)\leq \inf_{y\in \mathcal{Y}}\sup_{x\in \mathcal{X}}f(x,y).
$$
The seminal work Neumann [1928] shows that this inequality holds as an equality for matrix games where  $\mathcal{X} \subseteq \mathbb{R}^m$ ,  $\mathcal{Y} \subseteq \mathbb{R}^n$  are probability simplexes and we have  $f(x, y) = x^\top Ay$  given a payoff matrix  $A \in \mathbb{R}^{m \times n}$ . The result later gets generalized to the setting where  $\mathcal{X}, \mathcal{Y}$  are compact sets,  $f(x, \cdot)$  is quasi-convex for all  $x \in \mathcal{X}$ , and  $f(\cdot, y)$  is quasi-concave for all  $y \in \mathcal{Y}$  [Fan, 1953, Sion, 1958]. Much more recently, Yang et al. [2020] establishes the minimax equality when  $f$  satisfies the two-sided PL condition. For arbitrary functions  $f$ , the minimax equality need not be valid.
The validity of the minimax equality is essentially equivalent to the existence of a global Nash equilibrium  $(x^{\star},y^{\star})$  such that
$$
f (x, y ^ {\star}) \leq f (x ^ {\star}, y ^ {\star}) \leq f (x ^ {\star}, y), \quad \forall x \in \mathcal {X}, y \in \mathcal {Y}.
$$
The Nash equilibrium  $(x^{\star},y^{\star})$  is a point where neither player can improve their objective function value by changing its strategy. In general nonconvex-nonconcave settings where the global Nash equilibrium may not exist, alternative approximate local/global optimality notions are considered [Daskalakis and Panageas, 2018, Nouiehed et al., 2019, Adolphs et al., 2019, Jin et al., 2020].
Robust Reinforcement Learning. Robust RL studies finding the optimal policy in the worst-case scenario under environment uncertainty and/or possible adversarial attacks. Various robust RL models have been considered in the existing literature, such as: 1) the learning agent operates under uncertainty in the transition probability kernel [Goyal and Grand-Clement, 2022, Li et al., 2022, Panaganti and Kalathil, 2022, Wang et al., 2023], 2) an adversary exists and plays a two-player zero-sum Markov game against the learning agent [Pinto et al., 2017, Tessler et al., 2019], 3) the adversary does not affect the state transition but may manipulate the state observation [Havens et al., 2018, Zhang et al., 2020], 4) there is uncertainty or attack only on the reward [Wang et al., 2020, Banihashem et al., 2021, Sarkar et al., 2022], 5) the learning agent defends against attacks from a population of adversaries rather than a single one [Vinitsky et al., 2020]. A particular attack and defense model considered later in our paper is adapted from Banihashem et al. [2021].
Other Works on Connected Level Sets in Machine Learning. Last but not least, we note that our paper is related to the works that study the connectedness of the sublevel sets for the LQR optimization problem [Fatkhullin and Polyak, 2021] and for deep supervised learning under a regression loss [Nguyen, 2019]. The neural network architecture considered in our paper is inspired by and similar to the one in Nguyen [2019]. However, our result and analysis on deep RL are novel and significantly more challenging to establish, since 1) the underlying loss function in Nguyen [2019] is convex, while ours is a non-convex policy optimization objective, 2) the analysis of Nguyen [2019] relies critically on the assumption that the activation functions are uniquely invertible, while we use a non-uniquely invertible softmax activation function to generate policies within the probability simplex.
1.3 Connection between Gradient Domination and Connected Superlevel Sets
We loosely use the term "gradient domination" to indicate that a differentiable function does not have any sub-optimal stationary points. In this section, we use two examples to show that the gradient domination condition in general does not imply or get implied by the connectedness of the superlevel sets. The first example is a function that observes the gradient domination condition but has a disconnected set of maximizers (which implies that the superlevel is not always connected).
Consider  $f:[-4,4]\times [-2,0]\to \mathbb{R}$
$$
f (x, y) = \left\{ \begin{array}{l l} f _ {1} (x, y) = - (x - 1) ^ {3} + 3 (x - 1) - y ^ {2} - 2 y - 0. 0 2 (y + 1 0) ^ {2} (1 0 - x ^ {2}), \text {f o r} x \geq 0 \\ f _ {2} (x, y) = - (- x - 1) ^ {3} + 3 (- x - 1) - y ^ {2} - 2 y - 0. 0 2 (y + 1 0) ^ {2} (1 0 - x ^ {2}), \text {e l s e} \end{array} \right.
$$
It is obvious that the function is symmetric along the line  $x = 0$  and that  $f_{1}(0,y) = f_{2}(0,y)$  for all  $y\in [-2,0]$ . Computing the derivatives of  $f_{1}$  and  $f_{2}$  with respect to  $x$ , we have
$$
\nabla_ {x} f _ {1} (x, y) = - 3 (x - 1) ^ {2} + 3 + 0. 0 4 x (y + 1 0) ^ {2},
$$
$$
\nabla_ {x} f _ {2} (x, y) = 3 (x + 1) ^ {2} - 3 + 0. 0 4 x (y + 1 0) ^ {2}.
$$
We can again verify  $\nabla_x f_1(0,y) = \nabla_x f_2(0,y)$  for all  $y$ , which implies that the function  $f$  is everywhere continuous and differentiable. Visualization of  $f$  in Fig. 1 along with simple calculation (solving the system of equations  $\nabla_x f(x,y) = 0$  and  $\nabla_y f(x,y) = 0$ ) show that there are only two stationary points of  $f$  on  $[-4,4] \times [-2,0]$ . The two stationary points are  $(3.05, -1.12)$  and  $(-3.05, -1.12)$ , and they are both global maximizers on this domain, which means that the gradient domination condition is observed. However, the set of maximizers  $\{(3.05, -1.12), (-3.05, -1.12)\}$  is clearly disconnected.
We next present a function that has connected superlevel sets at all level but does not observe the gradient domination condition (i.e. has sub-optimal stationary points).
Figure 1: Visualization of Functions  $f$  (Left) and  $g$  (Right)
Consider  $g:\mathbb{R}^2\to \mathbb{R}$  defined as
$$
g (x, y) = - \left(x ^ {2} + y ^ {2}\right) ^ {2} + 4 \left(x ^ {2} + y ^ {2}\right).
$$
This is a volcano-shaped function, which we visualize in Fig. 1. It is obvious the superlevel set  $\{(x,y):g(x,y)\geq \lambda \}$  is always either a 2D circle (convex set) or a donut-shaped connected set depending on the choice of  $\lambda$ . However, the gradient domination condition does not hold as  $(0,0)$  is a first-order stationary point but not a global maximizer (it is actually a local minimizer).
Outline of the paper. The rest of the paper is organized as follows. In Section 2, we discuss the policy optimization problem in the tabular setting and establish the connectedness of the superlevel sets. Section 3 generalizes the result to a class of policies represented by over-parameterized neural networks. We introduce the structure of the neural network and the definition of super level sets in this context, and present our theoretical result. In Section 4, we use our main results on superlevel sets to derive two minimax theorems for robust RL. Finally, we conclude in Section 5 with remarks on future directions.
2 Connected Superlevel Set Under Tabular Policy
We consider the infinite-horizon average-reward MDP characterized by  $\mathcal{M} = (\mathcal{S},\mathcal{A},\mathcal{P},r)$ . We use  $\mathcal{S}$  and  $\mathcal{A}$  to denote the state and action spaces, which we assume are finite. The transition probability kernel is denoted by  $\mathcal{P}:S\times \mathcal{A}\to \Delta_{\mathcal{S}}$ , where  $\Delta_S$  denotes the probability simplex over  $\mathcal{S}$ . The reward function  $r:S\times \mathcal{A}\rightarrow [0,U_r]$  is bounded for some positive constant  $U_{r}$  and can also be regarded as a vector in  $\mathbb{R}^{|S|\times |\mathcal{A}|}$ . We use  $P^{\pi}\in \mathbb{R}^{S\times S}$  to represent the state transition probability matrix under policy  $\pi \in \Delta_{\mathcal{A}}^{\mathcal{S}}$ , where  $\Delta_A^S$  is the collection of probability simplexes over  $\mathcal{A}$  across the state space
$$
P _ {s ^ {\prime}, s} ^ {\pi} = \sum_ {a \in \mathcal {A}} \mathcal {P} \left(s ^ {\prime} \mid s, a\right) \pi (a \mid s), \quad \forall s ^ {\prime}, s \in \mathcal {S}. \tag {1}
$$
We consider the following ergodicity assumption in the rest of the paper, which is commonly made in the RL literature [Wang, 2017, Wei et al., 2020, Wu et al., 2020].
Assumption 1 Given any policy  $\pi$ , the Markov chain formed under the transition probability matrix  $P^{\pi}$  is ergodic, i.e. irreducible and aperiodic.
Let  $\mu_{\pi} \in \Delta_{\mathcal{S}}$  denote the stationary distribution of the states induced by policy  $\pi$ . As a consequence of Assumption 1, the stationary distribution  $\mu_{\pi}$  is unique and uniformly bounded away from 0 under any  $\pi$ . In addition,  $\mu_{\pi}$  is the unique eigenvector of  $P^{\pi}$  with the associated eigenvalue equal to 1, i.e.  $\mu_{\pi} = P^{\pi} \mu_{\pi}$ . Let  $\widehat{\mu}_{\pi} \in \Delta_{\mathcal{S} \times \mathcal{A}}$  denote the state-action stationary distribution induced by  $\pi$ , which can be expressed as
$$
\widehat {\mu} _ {\pi} (s, a) = \mu_ {\pi} (s) \pi (a \mid s). \tag {2}
$$
We measure the performance of a policy  $\pi$  under reward function  $r$  by the average cumulative reward  $J_{r}(\pi)$
$$
J _ {r} (\pi) \triangleq \lim _ {K \to \infty} \frac {\sum_ {k = 0} ^ {K} r (s _ {k} , a _ {k})}{K} = \mathbb {E} _ {s \sim \mu_ {\pi}, a \sim \pi} [ r (s _ {k}, a _ {k}) ] = \sum_ {s, a} r (s, a) \widehat {\mu} _ {\pi} (s, a).
$$
The objective of the policy optimization problem is to find the policy  $\pi$  that maximizes the average cumulative reward
$$
\max  _ {\pi \in \Delta_ {A} ^ {S}} J _ {r} (\pi). \tag {3}
$$
The superlevel set of  $J_{r}$  is the set of policies that achieve a value function greater than or equal to a specified level. Formally, given  $\lambda \in \mathbb{R}$ , the  $\lambda$ -superlevel set (or superlevel set) under reward  $r$  is defined as
$$
\mathcal {U} _ {\lambda , r} \triangleq \left\{\pi \in \Delta_ {\mathcal {A}} ^ {\mathcal {S}} \mid J _ {r} (\pi) \geq \lambda \right\}.
$$
The main focus of this section is to study the connectedness of this set  $\mathcal{U}_{\lambda, r}$ , which requires us to formally define a connected set.
Definition 1 A set  $\mathcal{U}$  is connected if for any  $x, y \in \mathcal{U}$  there exists a continuous map  $p: [0,1] \to \mathcal{U}$  such that  $p(0) = x$  and  $p(1) = y$ .
We say that a function is connected if its superlevel sets are connected at all levels. We also introduce the definition of equiconnected functions.
Definition 2 Given two spaces  $\mathcal{X}$  and  $\mathcal{Y}$ , the collection of functions  $\{f_y : \mathcal{X} \to \mathbb{R}\}_{y \in \mathcal{Y}}$  is said to be equiconnected if for every  $x_1, x_2 \in \mathcal{X}$ , there exists a continuous path map  $p : [0,1] \to \mathcal{X}$  such that
$$
p (0) = x _ {1}, \quad p (1) = x _ {2}, \quad f _ {y} (p (\alpha)) \geq \min  \{f _ {y} (x _ {1}), f _ {y} (x _ {2}) \},
$$
for all  $\alpha \in [0,1]$  and  $y\in \mathcal{V}$
Conceptually, the collection of functions  $\{f_y:\mathcal{X}\to \mathbb{R}\}_{y\in \mathcal{Y}}$  being equiconnected requires 1) that  $f_{y}(\cdot)$  is a connected function for all  $y\in \mathcal{V}$  (or equivalently, the set  $\{x\in \mathcal{X}:f_y(x)\geq \lambda \}$  is connected for all  $\lambda \in \mathbb{R}$  and  $y\in \mathcal{V}$ ) and 2) that the path map constructed to prove the connectedness of  $\{x\in \mathcal{X}:f_y(x)\geq \lambda \}$  is independent of  $y$ .
We now present our first main result of the paper, which states that the superlevel set  $\mathcal{U}_{\lambda,r}$  is always connected.
Theorem 1 Under Assumption 1, the superlevel set  $\mathcal{U}_{\lambda,r}$  is connected for any  $\lambda \in \mathbb{R}$  and  $r \in \mathbb{R}^{|S||A|}$ . In addition, the collection of functions  $\{J_r(\cdot):\Delta_A^S\to \mathbb{R}\}_{r\in \mathbb{R}^{|S|\times |A|}}$  is equiconnected.
Our result here extends easily to the infinite-horizon discounted-reward setting since a discounted-reward MDP can be regarded as an average-reward one with a slightly modified transition kernel [Konda, 2002].
The claim in Theorem 1 on the equiconnectedness of  $\{J_r\}_{r \in \mathbb{R}^{|S| \times |A|}}$  is a slightly stronger result than the connectedness of  $\mathcal{U}_{\lambda, r}$ , and plays an important role in the application to minmax theorems discussed later in Section 4.
We note that the proof, presented in Section A.1 of the appendix, mainly leverages the fact that the value function  $J_{r}(\pi)$  is linear in the state-action stationary distribution  $\widehat{\mu}_{\pi}$  and that there is a special connection (though nonlinear and nonconvex) between  $\widehat{\mu}_{\pi}$  and the policy  $\pi$ , which we take advantage
of to construct the continuous path map for the analysis. Specifically, given two policies  $\pi_1, \pi_2$  with  $J_r(\pi_1), J_r(\pi_2) \geq \lambda$ , we show that the policy  $\pi_\alpha$  defined as
$$
\pi_ {\alpha} (a \mid s) = \frac {\alpha \mu_ {\pi_ {1}} (s) \pi_ {1} (a \mid s) + (1 - \alpha) \mu_ {\pi_ {2}} (s) \pi_ {2} (a \mid s)}{\alpha \mu_ {\pi_ {1}} (s) + (1 - \alpha) \mu_ {\pi_ {2}} (s)}, \quad \forall \alpha \in [ 0, 1 ]
$$
is guaranteed to achieve  $J_{r}(\pi_{\alpha}) \geq \lambda$  for all  $\alpha \in [0,1]$ .
Besides playing a key role in the proof of Theorem 1, our construction of this path map may inform the design of algorithms in the future. Given any two policies with a certain guaranteed performance, we can generate a continuum of policies at least as good. As a consequence, if we find two optimal policies (possibly by gradient descent from different initializations) we can generate a range of interpolating optimal policies. If the agent has a preference over these policy (for example, to minimize certain energy like in  $H_{1}$  control, or if some policies are easier to implement physically), then the selection can be made on the continuum of optimal policies, which eventually leads to a more preferred policy.
3 Connected Superlevel Set Under Neural Network Parameterized Policy
In real-world reinforcement learning applications, it is common to use a deep neural network to parameterize the policy [Silver et al., 2016, Arulkumaran et al., 2017]. In this section, we consider the policy optimization problem under a special class of policies represented by an over-parameterized neural network and show that this problem still enjoys the important structure — the connectedness of the superlevel sets — despite the presence of the highly complex function approximation. Illustrated in Fig. 2, the neural network parameterizes the policy in a very natural manner which matches how neural networks are actually used in practice.
Figure 2: Neural Network Policy Representation
Mathematically, the parameterization can be described as follows. Each state  $s \in S$  is associated with a feature vector  $\phi(s) \in \mathbb{R}^d$ , which in practice is usually carefully selected to summarize the key information of the state. For state identifiability, we assume that the feature vector of each state is unique, i.e.
$$
\phi (s) \neq \phi \left(s ^ {\prime}\right), \quad \forall s, s ^ {\prime} \in \mathcal {S} \text {a n d} s \neq s ^ {\prime}.
$$
To map a feature vector  $\phi(s)$  to a policy distribution over state  $s$ , we employ a  $L$ -layer neural network, which in the  $k_{\mathrm{th}}$  layer has weight matrix  $W_k \in \mathbb{R}^{n_{k-1} \times n_k}$  and bias vector  $b_k \in \mathbb{R}^{n_k}$  with  $n_0 = d$  and  $n_L = |\mathcal{A}|$ . For the simplicity of notation, we use  $\Omega_k$  to denote the space of weight and bias parameters  $(W_k, b_k)$  of layer  $k$ , and we write  $\Omega = \Omega_1 \times \dots \times \Omega_L$ . We use  $\theta$  to denote the collection of the weights and biases
$$
\theta = \left(\left(W _ {1}, b _ {1}\right), \dots , \left(W _ {L}, b _ {L}\right)\right) \in \Omega
$$
We use the same activation function for layers 1 through  $L - 1$ , denoted by  $\sigma : \mathbb{R} \to \mathbb{R}$ , applied in an element-wise fashion to vectors. To ensure that the output of the neural network is a valid probability distribution, the activation function for the last layer is a softmax function, denoted by  $\psi : \mathbb{R}^{|A|} \to \Delta_A$ , i.e. for any vector  $v \in \mathbb{R}^{|A|}$
$$
\psi (v) _ {i} = \frac {\exp (v _ {i})}{\sum_ {i ^ {\prime} = 1} ^ {| \mathcal {A} |} \exp (v _ {i ^ {\prime}})}, \quad \forall i = 1, \dots , | \mathcal {A} |.
$$
With  $v \in \mathbb{R}^d$  as the input to a neural network with parameters  $\theta$ , we use  $f_k^\theta(v) \in \mathbb{R}^{n_k}$  to denote the output of the network at layer  $k$ . For  $k = 1, \dots, L$ ,  $f_k^\theta(v)$  is computed as