SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f _ {k} ^ {\theta} (v) = \left\{ \begin{array}{l l} \sigma \left(W _ {1} ^ {\top} v + b _ {1}\right) & k = 1 \\ \sigma \left(W _ {k} ^ {\top} f _ {k - 1} (v) + b _ {k}\right) & k = 2, 3, \dots , L - 1 \\ \psi \left(W _ {L} ^ {\top} f _ {L - 1} (v) + b _ {L}\right) & k = L. \end{array} \right. \tag {4}
$$
The policy  $\pi_{\theta} \in \mathbb{R}^{|S| \times |\mathcal{A}|}$  parametrized by  $\theta$  is the output of the final layer:
$$
\pi_ {\theta} (\cdot \mid s) = f _ {L} ^ {\theta} (\phi (s)) \in \Delta_ {\mathcal {A}}, \quad \forall s \in \mathcal {S}.
$$
Our analysis relies two assumptions about the structure of the neural network. The first concerns the invertibility of  $\sigma(\cdot)$  as well as the continuity and uniqueness of its inverse, which can be guaranteed by the following:
Assumption 2  $\sigma$  is strictly monotonic and  $\sigma (\mathbb{R}) = \mathbb{R}$ . In addition, there do not exist non-zero scalars  $\{p_i,q_i\}_{i = 1}^m$  with  $q_{i}\neq q_{j}$ $\forall i\neq j$  such that for some  $m > 0$ ,  $\sigma (x) = \sum_{i = 1}^{m}p_{i}\sigma (x - q_{i})$ ,  $\forall x\in \mathbb{R}$ .
We note that this assumption holds for common activation functions including leaky-ReLU and parametric ReLU [Xu et al., 2015].
Our second assumption is that the neural network is sufficiently over-parameterized and that the number of parameters decreases with each layer.
Assumption 3 The output of the first layer is wider than  $2|\mathcal{S}|$ , and the width of the network decreases over the layers, i.e.
$$
n _ {1} \geq 2 | \mathcal {S} |, a n d n _ {1} > n _ {2} > \dots > n _ {L} = | \mathcal {A} |.
$$
Neural networks meeting this criteria have a number of weight parameters that is larger than the cardinality of the state space, making them impractical for large  $|S|$ . While ongoing work seeks to relax or remove this assumption, we point out that similar over-parameterization assumptions are critical and very common in most existing works on the theory of neural networks [Zou and Gu, 2019, Nguyen, 2019, Liu et al., 2022, Martinetz and Martinetz, 2022, Pandey and Kumar, 2023].
The  $\lambda$ -superlevel set of the value function with respect to  $\theta$  under reward function  $r$  is
$$
\mathcal {U} _ {\lambda , r} ^ {\Omega} \triangleq \left\{\theta \in \Omega \mid J _ {r} \left(\pi_ {\theta}\right) \geq \lambda \right\}.
$$
Our next main theoretical result guarantees the connectedness of  $\mathcal{U}_{\lambda,r}^{\Omega}$ .
Theorem 2 Under Assumptions 1-3, the superlevel set  $\mathcal{U}_{\lambda,r}^{\Omega}$  is connected for any  $\lambda \in \mathbb{R}$ . In addition, with  $J_{r,\Omega}(\theta) \triangleq J_r(\pi_\theta)$ , the collection of functions  $\{J_{r,\Omega}(\cdot):\Omega \to \mathbb{R}\}_{r\in \mathbb{R}^{|S|\times |\mathcal{A}|}}$  is equiconnected.
The proof of this theorem is deferred to the appendix. Similar to Theorem 1, the claim in Theorem 2 on the equiconnectedness of  $\{J_{r,\Omega}\}_{r\in \mathbb{R}^{|S|\times |\mathcal{A}|}}$  is again stronger than the connectedness of  $\mathcal{U}_{\lambda ,r}^{\Omega}$  and needs to be derived for the application to minimax theorems, which we discuss in the next section.
4 Application to Robust Reinforcement Learning
In this section, we consider the robust RL problem under adversarial reward attack, which can be formulated as a convex-nonconcave minimax optimization program. In Section 4.1, we show that the minimax equality holds for this optimization program in the tabular policy setting and under policies represented by a class of neural networks, as a consequence of our results in Sections 2 and 3. To our best knowledge, the existence of the Nash equilibrium for this robust RL problem has not been established before even in the tabular case. A specific example of this type of robust RL problems is given in Section 4.2.
4.1 Minimax Theorem
Robust RL in general studies identifying a policy with reliable performance under uncertainty or attacks. A wide range of formulations have been proposed for robust RL (which we reviewed in details in Section 1.2), and an important class of formulations takes the form of defending against an adversary that can modify the reward function in a convex manner. Specifically, the objective of the learning agent can be described as solving the following minimax optimization problem
$$
\max  _ {\pi \in \Delta_ {A} ^ {S}} \min  _ {r \in \mathcal {C}} J _ {r} (\pi), \tag {5}
$$
where  $\mathcal{C}$  is some convex set. It is unclear from the existing literature whether minimax inequality holds for this problem, i.e.
$$
\max  _ {\pi \in \Delta_ {A} ^ {S}} \min  _ {r \in \mathcal {C}} J _ {r} (\pi) = \min  _ {r \in \mathcal {C}} \max  _ {\pi \in \Delta_ {A} ^ {S}} J _ {r} (\pi), \tag {6}
$$
and we provide a definitive answer to this question. We note that there exists a classic minimax theorem on a special class of convex-nonconcave functions [Simons, 1995], which we adapt and simplify as follows.
Theorem 3 Consider a separately continuous function  $f: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ , with  $\mathcal{Y}$  being a convex, compact set. Suppose that  $f(x, \cdot)$  is convex for all  $x \in \mathcal{X}$ . Also suppose that the collection of functions  $\{f(\cdot, y)\}_{y \in \mathcal{Y}}$  is equiconnected. Then, we have
$$
\sup  _ {x \in \mathcal {X}} \min  _ {y \in \mathcal {Y}} f (x, y) = \min  _ {y \in \mathcal {Y}} \sup  _ {x \in \mathcal {X}} f (x, y). \tag {7}
$$
Theorem 3 states that the minimax equality holds under two main conditions (besides the continuity condition, which can easily be verified to hold for  $J_{r}(\pi)$ ). First, the function  $f(x,y)$  needs to be convex with respect to the variable  $y$  within a convex, compact constraint set. Second,  $f(x,y)$  needs to have a connected superlevel set with respect to  $x$ , and the path function constructed to prove the connectedness of the superlevel set is independent of  $y$ . As we have shown in this section and earlier in the paper, if we model  $J_{r}(\pi)$  by  $f(x,y)$  with  $\pi$  and  $r$  corresponding to  $x$  and  $y$ , both conditions are observed by the optimization problem (5), which allows us to state the following corollary.
Corollary 1 Suppose that the Markov chain  $\mathcal{M}$  satisfies Assumption 1 on ergodicity. Then, the minimax equality (6) holds.
When the neural network presented in Section 3 is used to represent the policy, the collection of functions  $\{J_{r,\Omega}\}_{r}$  is also equiconnected. This allows us to extend the minimax equality above to the neural network policy class. Specifically, consider problem (5) where the policy  $\pi_{\theta}$  is represented by the parameter  $\theta \in \Omega$  as described in Section 3. Using  $f(x,y)$  to model  $J_{r}(\pi_{\theta})$  with  $x$  and  $y$  mirroring  $\theta$  and  $r$ , we can easily establish the minimax theorem in this case as a consequence of Theorem 2 and 3.
Corollary 2 Suppose that the Markov chain  $\mathcal{M}$  satisfies Assumption 1 on ergodicity and that the neural policy class satisfies Assumptions 2-3. Then, we have
$$
\sup  _ {\theta \in \Omega} \min  _ {r \in \mathcal {C}} J _ {r} (\pi_ {\theta}) = \min  _ {r \in \mathcal {C}} \sup  _ {\theta \in \Omega} J _ {r} (\pi_ {\theta}). \tag {8}
$$
Corollary 1 and 2 establish the minimax equality (or equivalently, the existence of the Nash equilibrium) for the robust reinforcement learning problem under adversarial reward attack for the tabular and neural network policy class, respectively. To our best knowledge, these results are both novel and previously unknown in the existing literature. The Nash equilibrium is an important global optimality notion in minimax optimization, and the knowledge on its existence can provide strong guidance on designing and analyzing algorithms for solving the problem.
4.2 Example - Defense Against Reward Poisoning
We now discuss a particular example of (5). We consider the infinite horizon, average reward MDP  $\mathcal{M} = (\mathcal{S},\mathcal{A},\mathcal{P},r)$  introduced in Section 2, where  $r$  is the true, unpoisoned reward function. Let  $\Pi^{\mathrm{det}}$  denote the set of deterministic policies from  $\mathcal{S}$  to  $\mathcal{A}$ . With the perfect knowledge of this MDP, an attacker has a target policy  $\pi_{\dagger}\in \Pi^{\mathrm{det}}$  and tries to make the learning agent adopt the policy by
manipulating the reward function. Mathematically, the goal of the attacker can be described by the function  $\operatorname{Attack}(r, \pi_{\dagger}, \epsilon_{\dagger})$  which returns a poisoned reward under the true reward  $r$ , the target policy  $\pi_{\dagger}$ , and a pre-selected margin parameter  $\epsilon_{\dagger} \geq 0$ .  $\operatorname{Attack}(r, \pi_{\dagger}, \epsilon_{\dagger})$  is the solution to the following optimization problem
$$
\operatorname {A t t a c k} \left(r, \pi_ {\dagger}, \epsilon_ {\dagger}\right) = \underset {r ^ {\prime}} {\operatorname {a r g m i n}} \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} \left(r ^ {\prime} (s, a) - r (s, a)\right) ^ {2} \tag {9}
$$
$$
\begin{array}{l l} \text {s . t .} & J _ {r ^ {\prime}} (\pi_ {\dagger}) \geq J _ {r ^ {\prime}} (\pi) + \epsilon_ {\dagger}, \quad \forall \pi \in \Pi^ {\det } \backslash \pi_ {\dagger}. \end{array}
$$
In other words, the attacker needs to minimally modify the reward function to make  $\pi_{\dagger}$  the optimal policy under the poisoned reward. This optimization program minimizes a quadratic loss under a finite number of linear constraints and is obviously convex.
The learning agent observes the poisoned reward  $r_{\dagger} = \text{Attack}(r, \pi_{\dagger}, \epsilon_{\dagger})$  rather than the original reward  $r$ . As noted in Banihashem et al. [2021], without any defense, the learning agent solves the policy optimization problem under  $r_{\dagger}$  to find  $\pi_{\dagger}$ , which may perform arbitrarily badly under the original reward. One way to defend against the attack is to maximize the performance of the agent in the worst possible case of the original reward, which leads to solving a minimax optimization program of the form
$$
\max  _ {\pi \in \Delta_ {A} ^ {S}} \min  _ {r ^ {\prime}} J _ {r ^ {\prime}} (\pi) \quad \text {s . t .} \operatorname {A t t a c k} \left(r ^ {\prime}, \pi_ {\dagger}, \epsilon_ {\dagger}\right) = r _ {\dagger}. \tag {10}
$$
When the policy  $\pi$  is fixed, (10) reduces to
$$
\min  _ {r ^ {\prime}} J _ {r ^ {\prime}} (\pi) \quad \text {s . t .} \operatorname {A t t a c k} \left(r ^ {\prime}, \pi_ {\dagger}, \epsilon_ {\dagger}\right) = r _ {\dagger}. \tag {11}
$$
With the justification deferred to Appendix D, we point out that (11) consists of a linear objective function and a convex (and compact) constraint set, and is therefore a convex program. On the other side, when we fix the reward  $r'$ , (10) reduces to a standard policy optimization problem.
We are interested in investigating whether the following minimax equality holds.
$$
\max  _ {\pi \in \Delta_ {A} ^ {S}} \min  _ {r ^ {\prime}: \operatorname {A t t a c k} \left(r ^ {\prime}, \pi_ {\dagger}, \epsilon_ {\dagger}\right) = r _ {\dagger}} J _ {r ^ {\prime}} (\pi) = \min  _ {r ^ {\prime}: \operatorname {A t t a c k} \left(r ^ {\prime}, \pi_ {\dagger}, \epsilon_ {\dagger}\right) = r _ {\dagger}} \max  _ {\pi \in \Delta_ {A} ^ {S}} J _ {r ^ {\prime}} (\pi). \tag {12}
$$
This is a special case of (5) with  $\mathcal{C} = \{r' \mid \text{Attack}(r', \pi_{\dagger}, \epsilon_{\dagger}) = r_{\dagger}\}$ , which can be verified to be a convex set. Therefore, the validity of (12) directly follows from Corollary 1. Similarly, in the setting of neural network parameterized policy we can establish
$$
\max  _ {\theta \in \Omega} \min  _ {r ^ {\prime}: \operatorname {A t t a c k} (r ^ {\prime}, \pi_ {\dagger}, \epsilon_ {\dagger}) = r _ {\dagger}} J _ {r ^ {\prime}} (\pi_ {\theta}) = \min  _ {r ^ {\prime}: \operatorname {A t t a c k} (r ^ {\prime}, \pi_ {\dagger}, \epsilon_ {\dagger}) = r _ {\dagger}} \max  _ {\theta \in \Omega} J _ {r ^ {\prime}} (\pi_ {\theta})
$$
as a result of Corollary 2.
5 Conclusions & Future Work
We study the superlevel set of the policy optimization problem under the MDP framework and show that it is always a connected set under a tabular policy and for policies parameterized by a class of neural networks. We apply this result to derive a previously unknown minimax theorem for a robust RL problem. An immediate future direction of the work is to investigate whether/how the result discussed in this paper can be used to design better RL algorithms. In Fatkhullin and Polyak [2021], the authors show that the original LQR problem has connected level sets, but the partially observable LQR does not. It is interesting to study whether this observation extends to the MDP setting, i.e. the policy optimization problem under a partially observable MDP can be shown to have disconnected superlevel sets.
This work was supported in part by the NSF AI Institute AI4OPT, NSF 2112533.
Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6202-6209, 2020.
Mengdi Wang. Primal-dual  $\pi$  learning: Sample complexity and sublinear run time for ergodic markov decision problems. arXiv preprint arXiv:1710.06100, 2017.
Yue Wang, Alvaro Velasquez, George Atia, Ashley Prater-Bennette, and Shaofeng Zou. Robust average-reward markov decision processes. arXiv preprint arXiv:2301.00858, 2023.
Chen-Yu Wei, Mehdi Jafarnia Jahromi, Haipeng Luo, Hiteshi Sharma, and Rahul Jain. Model-free reinforcement learning in infinite-horizon average-reward markov decision processes. In International conference on machine learning, pages 10170-10180. PMLR, 2020.
Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems, 33:17617-17628, 2020.
Lin Xiao. On the convergence rates of policy gradient methods. The Journal of Machine Learning Research, 23(1):12887-12922, 2022.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853, 2015.
Junchi Yang, Negar Kiyavash, and Niao He. Global convergence and variance reduction for a class of nonconvex-nonconcave minimax problems. Advances in Neural Information Processing Systems, 33:1153-1165, 2020.
Zhuoran Yang, Yongxin Chen, Mingyi Hong, and Zhaoran Wang. Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost. Advances in neural information processing systems, 32, 2019.
Sihan Zeng, Malik Aqeel Anwar, Thinh T Doan, Arijit Raychowdhury, and Justin Romberg. A decentralized policy gradient approach to multi-task reinforcement learning. In Uncertainty in Artificial Intelligence, pages 1002-1012. PMLR, 2021a.
Sihan Zeng, Thinh T Doan, and Justin Romberg. A two-time-scale stochastic optimization framework with applications in control and reinforcement learning. arXiv preprint arXiv:2109.14756, 2021b.
Sihan Zeng, Thinh Doan, and Justin Romberg. Regularized gradient descent ascent for two-player zero-sum markov games. Advances in Neural Information Processing Systems, 35:34546-34558, 2022.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:21024-21037, 2020.
Difan Zou and Quanquan Gu. An improved analysis of training over-parameterized deep neural networks. Advances in neural information processing systems, 32, 2019.
A Proof of Theorems
A.1 Proof of Theorem 1:
We note that there exists a bijective map between  $\pi$  and  $\widehat{\mu}_{\pi}$  where  $\widehat{\mu}_{\pi}$  is induced by  $\pi$  according to (2) and conversely
$$
\pi (a \mid s) = \frac {\widehat {\mu} _ {\pi} (s , a)}{\mu_ {\pi} (s)} = \frac {\widehat {\mu} _ {\pi} (s , a)}{\sum_ {a \in \mathcal {A}} \widehat {\mu} _ {\pi} (s , a)}, \tag {13}
$$
provided that  $\mu_{\pi}(s) \neq 0$ , which is guaranteed by Assumption 1. Eq. (13) inspires the construction of the path map.
To prove that the superlevel set is connected, we show that for any  $\lambda \in \mathbb{R}$  and  $\pi_1, \pi_2 \in \mathcal{U}_{\lambda,r}$ , there exists a continuous path map  $p: [0,1] \to \mathcal{U}_{\lambda,r}$  such that  $p(0) = \pi_1$  and  $p(1) = \pi_2$ . We now construct the path function  $p$  by defining
$$
p (\alpha) (a \mid s) = \frac {\alpha \mu_ {\pi_ {1}} (s) \pi_ {1} (a \mid s) + (1 - \alpha) \mu_ {\pi_ {2}} (s) \pi_ {2} (a \mid s)}{\alpha \mu_ {\pi_ {1}} (s) + (1 - \alpha) \mu_ {\pi_ {2}} (s)},
$$
which is well-defined for all  $\alpha \in [0,1]$  as  $\mu_{\pi_1}(s),\mu_{\pi_2}(s)$  are positive for all  $s\in S$ . Note that the construction of  $p$  does not depend on the reward function  $r$ . It is easy to see that  $p(\alpha)\in \Delta_A^S$  is a continuous in  $\alpha$ . To stress that  $p(\alpha)$  is in the policy space, we denote  $\pi_{\alpha} = p(\alpha)$ .
Recall the definition of the transition probability matrix in (1). We define  $B \in \mathbb{R}^{|\mathcal{S}|}$  as
$$
B = P ^ {\pi_ {\alpha}} \cdot \left(\alpha \mu_ {\pi_ {1}} + (1 - \alpha) \mu_ {\pi_ {2}}\right).
$$
Each entry of  $B$  can be expressed as
$$
\begin{array}{l} B \left(s ^ {\prime}\right) = \sum_ {s, a} \mathcal {P} \left(s ^ {\prime} \mid s, a\right) \pi_ {\alpha} (a \mid s) (\alpha \mu_ {\pi_ {1}} (s) + (1 - \alpha) \mu_ {\pi_ {2}} (s)) \\ = \sum_ {s, a} \mathcal {P} \left(s ^ {\prime} \mid s, a\right) \frac {\alpha \mu_ {\pi_ {1}} (s) \pi_ {1} (a \mid s) + (1 - \alpha) \mu_ {\pi_ {2}} (s) \pi_ {2} (a \mid s)}{\alpha \mu_ {\pi_ {1}} (s) + (1 - \alpha) \mu_ {\pi_ {2}} (s)} (\alpha \mu_ {\pi_ {1}} (s) + (1 - \alpha) \mu_ {\pi_ {2}} (s)) \\ = \sum_ {s, a} \mathcal {P} \left(s ^ {\prime} \mid s, a\right) \alpha \mu_ {\pi_ {1}} (s) \pi_ {1} (a \mid s) + \sum_ {s, a} \mathcal {P} \left(s ^ {\prime} \mid s, a\right) (1 - \alpha) \mu_ {\pi_ {2}} (s) \pi_ {2} (a \mid s) \\ = \alpha \sum_ {s, a} P _ {s ^ {\prime}, s} ^ {\pi_ {1}} \mu_ {\pi_ {1}} (s) + (1 - \alpha) \sum_ {s, a} P _ {s ^ {\prime}, s} ^ {\pi_ {2}} \mu_ {\pi_ {2}} (s) \\ = \alpha \mu_ {\pi_ {1}} \left(s ^ {\prime}\right) + (1 - \alpha) \mu_ {\pi_ {2}} \left(s ^ {\prime}\right), \\ \end{array}
$$
$$
P ^ {\pi_ {\alpha}} \cdot \left(\alpha \mu_ {\pi_ {1}} + (1 - \alpha) \mu_ {\pi_ {2}}\right) = \alpha \mu_ {\pi_ {1}} + (1 - \alpha) \mu_ {\pi_ {2}}. \tag {14}
$$
A consequence of Assumption 1 is that for any policy  $\pi$  there is a unique eigenvector of  $P^{\pi}$  associated with the eigenvalue 1, and this eigenvector (properly normalized) is the stationary distribution. Therefore, (14) means that  $\alpha \mu_{\pi_1} + (1 - \alpha)\mu_{\pi_2}$  has to be the stationary distribution under policy  $\pi_{\alpha}$ , i.e.
$$
\mu_ {\pi_ {\alpha}} = \alpha \mu_ {\pi_ {1}} + (1 - \alpha) \mu_ {\pi_ {2}}.
$$
As a result, for all  $s \in S, a \in \mathcal{A}$
$$
\begin{array}{l} \widehat {\mu} _ {\pi_ {\alpha}} (s, a) = \mu_ {\pi_ {\alpha}} (s) \pi_ {\alpha} (a \mid s) \\ = \left(\alpha \mu_ {\pi_ {1}} (s) + (1 - \alpha) \mu_ {\pi_ {2}} (s)\right) \frac {\alpha \mu_ {\pi_ {1}} (s) \pi_ {1} (a \mid s) + (1 - \alpha) \mu_ {\pi_ {2}} (s) \pi_ {2} (a \mid s)}{\alpha \mu_ {\pi_ {1}} (s) + (1 - \alpha) \mu_ {\pi_ {2}} (s)} \\ = \alpha \mu_ {\pi_ {1}} (s) \pi_ {1} (a \mid s) + (1 - \alpha) \mu_ {\pi_ {2}} (s) \pi_ {2} (a \mid s) \\ = \alpha \widehat {\mu} _ {\pi_ {1}} (s, a) + (1 - \alpha) \widehat {\mu} _ {\pi_ {2}} (s, a). \\ \end{array}
$$
Note that  $J_{r}(\pi) = \sum_{s\in \mathcal{S},a\in \mathcal{A}}r(s,a)\widehat{\mu}_{\pi}(s,a)$ . Since  $\pi_{\pi_1},\pi_{\pi_2}\in \mathcal{U}_{\lambda ,r}$ , we know
$$
\sum_ {s \in \mathcal {S}, a \in \mathcal {A}} r (s, a) \widehat {\mu} _ {\pi_ {1}} (s, a) \geq \lambda , \quad \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} r (s, a) \widehat {\mu} _ {\pi_ {2}} (s, a) \geq \lambda .
$$
Therefore, we have for any  $\alpha \in [0,1]$
$$
J _ {r} (\pi_ {\alpha}) = \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} r (s, a) \widehat {\mu} _ {\pi_ {\alpha}} (s, a) = \sum_ {s \in \mathcal {S}, a \in \mathcal {A}} r (s, a) (\alpha \widehat {\mu} _ {\pi_ {1}} (s, a) + (1 - \alpha) \widehat {\mu} _ {\pi_ {2}} (s, a)) \geq \lambda ,
$$
which implies  $\pi_{\alpha} \in \mathcal{U}_{\lambda, r}$ . So far we have verified that the constructed path map  $p$  is indeed continuous and maps  $\alpha \in [0,1]$  to  $\mathcal{U}_{\lambda, r}$  with  $p(0) = \pi_1$  and  $p(1) = \pi_2$ . This concludes the proof on the connectedness of the superlevel set  $\mathcal{U}_{\lambda, r}$ . The claim on the equiconnectedness simply follows from the fact that the construction of the path map  $p$  does not depend on the reward function.
We use  $X$  to denote the concatenation of the feature vectors across all states
$$
X \triangleq \left[ \begin{array}{c} \phi (s _ {1}) ^ {\top} \\ \phi (s _ {2}) ^ {\top} \\ \vdots \\ \phi (s _ {| \mathcal {S} |}) ^ {\top} \end{array} \right] \in \mathbb {R} ^ {| \mathcal {S} | \times d}
$$
In the analysis we may apply the softmax function  $\psi$  to a matrix in a row-wise fashion. Specifically, for any  $n\geq 1$  and matrix  $M\in \mathbb{R}^{n\times |\mathcal{A}|}$ , we have
$$
\psi (M) _ {i, j} = \frac {\exp (M _ {i , j})}{\sum_ {j ^ {\prime} = 1} ^ {| \mathcal {A} |} \exp (M _ {i , j ^ {\prime}})} \quad \forall i = 1, \dots , n.
$$
The softmax operator  $\psi$  can be inverted up to an additive constant factor. We define  $\psi_{inv}$  for any matrix  $M\in \mathbb{R}^{n\times |\mathcal{A}|}$  as
$$
\psi_ {i n v} (M) _ {i, j} = \log (M _ {i, j}) + c _ {i} \quad \forall i, j,
$$
with  $c_{i}$  determined such that  $\sum_{j = 1}^{|A|}\psi_{inv}(M)_{i,j} = 0$ . Note that  $\psi_{inv}$  is a right inverse of  $\psi$ , i.e.  $\psi (\psi_{inv}(M)) = M$  for all matrix  $A$ .
When the input to a neural network with parameter  $\theta$  is the feature table  $X$ , we denote the output of layer  $k$  by  $F_{k}^{\theta}\in \mathbb{R}^{|\mathcal{S}|\times n_{k}}$ . According to (4),  $F_{k}^{\theta}$  can be expressed as
$$
F _ {k} ^ {\theta} = \left\{ \begin{array}{l l} \sigma (X W _ {1} + \mathbf {1} _ {| \mathcal {S} |} b _ {1} ^ {\top}) & k = 1 \\ \sigma (F _ {k - 1} ^ {\theta} W _ {k} + \mathbf {1} _ {| \mathcal {S} |} b _ {k} ^ {\top}) & k = 2, 3, \dots , L - 1 \\ \psi (F _ {L - 1} ^ {\theta} W _ {L} + \mathbf {1} _ {| \mathcal {S} |} b _ {L} ^ {\top}) & k = L \end{array} \right.
$$
where  $\mathbf{1}_{|\mathcal{S}|}$  is the all-one vector of dimension  $|\mathcal{S}|\times 1$ . Note that  $F_{L}^{\theta}\in \mathbb{R}^{|\mathcal{S}|\times |\mathcal{A}|}$  is the policy table produced by the neural network, i.e.  $\pi_{\theta} = F_{L}^{\theta}$ .
The proof of Theorem 2 relies on the following intermediate results, which we now present. The proof of Proposition 1 can be found in Appendix B.
Proposition 1 If  $\operatorname{rank}(X) = |\mathcal{S}|$ , then under Assumption 1 and 2, the superlevel set  $\mathcal{U}_{\lambda,r}^{\Omega}$  is connected for all  $\lambda \in \mathbb{R}$ .
Lemma 1 Let  $(X,W,b,V)\in \mathbb{R}^{|S|\times n_0}\times \mathbb{R}^{n_0\times n_1}\times \mathbb{R}^{n_1}\times \mathbb{R}^{n_1\times n_2}$ . Let  $Z = \sigma (XW + 1_{|S|b^{\top}})V$ . Suppose  $X$  has distinct rows. Then, under Assumption 2 and 3, there exists a continuous path map  $c:[0,1]\to \mathbb{R}^{n_0\times n_1}\times \mathbb{R}^{n_1}\times \mathbb{R}^{n_1\times n_2}$  with  $c(\lambda) = (W(\lambda),b(\lambda),V(\lambda))$  such that
1)  $c(0) = (W,b,V)$
2)  $\sigma\left(XW(\lambda) + \mathbf{1}_{|\mathcal{S}|}b(\lambda)^T\right)V(\lambda) = Z, \forall \lambda \in [0,1]$ ,
3) rank  $\left(\sigma\left(XW(1)+\mathbf{1}_{|S|}b(1)^T\right)\right)=N.$