SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Convergence of Actor-Critic Methods with Multi-Layer Neural Networks
Haoxing Tian, Ioannis Ch. Paschalidis, Alex Olshovsky
Boston, MA 02215, USA
{tianhx, yannisp, alexols}@bu.edu
The early theory of actor-critic methods considered convergence using linear function approximators for the policy and value functions. Recent work has established convergence using neural network approximators with a single hidden layer. In this work we are taking the natural next step and establish convergence using deep neural networks with an arbitrary number of hidden layers, thus closing a gap between theory and practice. We show that actor-critic updates projected on a ball around the initial condition will converge to a neighborhood where the average of the squared gradients is  $\tilde{O}(1/\sqrt{m}) + O(\epsilon)$ , with  $m$  being the width of the neural network and  $\epsilon$  the approximation quality of the best critic neural network over the projected set.
Reinforcement Learning (RL) has emerged as a powerful tool for solving decision-making problems in a model-free way. Among the various RL algorithms, the Actor-Critic (AC) method (Konda & Tsitsiklis (1999); Barto et al. (1983)) has shown great success in various domains, including robotics, game playing, and control systems (LeCun et al. (2015); Mnih et al. (2016); Silver et al. (2017)). AC involves simultaneous updates of two networks: an actor network that employs policy gradient (Sutton et al. (1999)) to update a parameterized policy, and a critic network which is driven by the Temporal Differences (TD) in the estimated value function. While AC methods with neural networks used for both actor and critic have achieved widespread use in practice, a fully satisfactory analysis of their convergence guarantees is currently lacking.
In recent years, a number of theoretical studies of AC have obtained provable convergence rates and performance analyses. Almost all works in this area assumed linear, rather than neural network-based, approximators for both actor and critic. A "two-timescale" linear AC was analysed in Wu et al. (2020), with a convergence rate of  $\tilde{O}(T^{-1/4})$ , where  $T$  is the total number of iterations and  $\tilde{O}(\cdot)$  refers to potential logarithmic terms omitted from the notation; the term "two-timescale" refers to the fact that the stepsizes for the actor update and critic update are not proportional to each other, but rather the actor steps are asymptotically negligible compared to the critic steps. A "single-timescale" linear AC method was considered in Olshovsky & Gharesifard (2022); Chen et al. (2021) and both works obtained a convergence rate of  $O(T^{-0.5})$  under an i.i.d. sampling assumption on the underlying MDP. The more realistic Markov sampling case was analyzed in the recent paper Chen & Zhao (2022), which also established a convergence rate of  $\tilde{O}(T^{-0.5})$ . All these results relied on linear approximations.
To our knowledge, convergence rates for AC with neural approximators were analyzed only in two recent works Wang et al. (2019); Cayci et al. (2022). Both of these papers considered neural net-
works with a single hidden layer. The paper Wang et al. (2019) obtained a convergence rate of  $O\left(T^{-0.5}\right)$  with a final error of  $O\left(m^{-0.25}\right)$  under i.i.d. sampling, where  $m$  is the width of hidden layer. The case of Markov sampling was considered in Cayci et al. (2022) which improved this to  $\tilde{O}\left(T^{-0.5}\right)$  and  $\tilde{O}\left(m^{-0.5}\right)$ , respectively. Further, both Wang et al. (2019); Cayci et al. (2022) considered "double-loop" methods where, in the inner loop, the critic takes sufficiently many steps to accurately estimate  $Q$ -values. Such double-loop methods do not match prevailing practice and are considerably easier to analyze since they can be shown to approximate gradient descent.
Further, Cayci et al. (2022) required a projection onto a ball of radius  $O(m^{-1/2})$  around the initial condition. Although a full representation theory for such neural networks is unknown, this is clearly limiting as compared to Wang et al. (2019) which only required projection onto a ball of constant radius. For nonlinear approximations, such projections are usually needed to stabilize the algorithm; without them, AC can diverge both in theory and practice.
Table 1: Comparisons with previous work.
Reference	Algorithm	Sampling	Approximation	Projection Radius	Convergence rate
w.r.t. T	w.r.t. m
Wu et al. (2020)	Two-timescale Single-loop	Markov	Linear	N/A	\(\tilde{O}\left(T^{-0.4}\right)\)	N/A
Olshevsky & Gharesifard (2022)	Single-timescale Single-loop	I.i.d.	Linear	N/A	\(O\left(T^{-0.5}\right)\)	N/A
Chen et al. (2021)	Single-timescale Single-loop	I.i.d'	Linear	N/A	\(O\left(T^{-0.5}\right)\)	N/A
Chen & Zhao (2022)	Single-timescale Single-loop	Markov	Linear	N/A	\(\tilde{O}\left(T^{-0.5}\right)\)	N/A
Wang et al. (2019)	Double-loop	I.i.d.	Single hidden layer	Constant	\(O\left(T^{-0.5}\right)\)	\(O\left(m^{-0.25}\right)\)
Cayci et al. (2022)	Double-loop	Markov	Single hidden layer	Decaying	\(\tilde{O}\left(T^{-0.5}\right)\)m sufficiently large	\(\tilde{O}\left(m^{-0.5}\right)\)
Ours	Single-timescale Single-loop	Markov	Any depth	Constant	\(\tilde{O}\left(T^{-0.5}\right)\)	\(\tilde{O}\left(m^{-0.5}\right)\)
The main contribution of this paper is to provide the first analysis of AC with neural networks of arbitrary depth. While replicating the earlier results of a  $\tilde{O}\left(T^{-0.5}\right)$  convergence rate and  $\tilde{O}\left(m^{-0.5}\right)$  error, our work considers a single-loop method with proportional step-sizes (sometimes called "single-timescale"). We prove this result under Markov sampling and project onto a ball of constant radius around the initial condition. An explicit comparison of our result to previous work is given in Table 1. A more technical comparison is also given later after the statement of our main result.
Our main technical tool is the so-called "gradient splitting" view of TD learning. This idea began with the paper Ollivier (2018) which observed that TD learning is exactly gradient descent when the underlying policy is such that the state transition matrix is reversible. In Liu & Olshovsky (2021), this was generalized to non-reversible policies by introducing the notion of a "gradient splitting" (discussed formally later in this work) and observing that, for linear approximation, TD updates are an example of gradient splitting. Gradient splitting is closely related to gradient descent, and the two processes can be analyzed similarly. A generalization to neural TD learning was given in Tian et al. (2023), which argued for an interpretation of nonlinear TD as approximate gradient splitting.
The analysis of AC that we perform in this work is trickier because both actor and critic updates rely on each other, and one must prove that the resulting errors in each process do not compound in interaction with each other. This difficulty arises because we do not consider the "double loop" case where the actor can effectively wait for the critic to converge, so that actor steps resemble gradient steps with error; rather both actor and critic update simultaneously their (imperfect) estimates. Similarly to what was done in Olshevsky & Gharesifard (2022), we show that we can draw on some ideas from control theory to prove that the resulting process converges with a so-called "small-gain" analysis.
We begin by standardizing notation and stating the key concepts that will enable us to formulate our results alongside all the assumptions they require.
2.1 Markov Decision Processes (MDP)
A finite discounted-reward MDP can be described by a tuple  $(S, A, P_{\mathrm{env}}, r, \gamma)$  where  $S$  is a finite state-space whose elements are vectors, and we use  $s_0 \in S$  to denote the starting state;  $A$  is a finite action space with cardinality  $n_a$ ;  $P_{\mathrm{env}} = (P_{\mathrm{env}}(s'|s,a))_{s,s' \in S, a \in A}$  is the transition probability matrix, where  $P_{\mathrm{env}}(s'|s,a)$  is the probability of transitioning from  $s$  to  $s'$  after taking action  $a$ ;  $r: S \times A \to \mathbb{R}$  is the reward function, where  $r(s,a)$  stands for the expected reward at state  $s$  and taking action  $a$ ; and  $\gamma \in (0,1)$  is the discount factor.
A policy  $\pi$  is a mapping  $\pi : S \times A \to [0,1]$  where  $\pi(a|s)$  is the probability that the agent takes action  $a$  in state  $s$ . Given a policy  $\pi$ , we can define the state transition matrix  $P_{\pi}' = (P_{\pi}'(s'|s))_{s,s' \in S}$  and the state-action transition matrix  $P_{\pi} = (P_{\pi}(s', a'|s, a))_{(s,a),(s',a') \in S \times A}$  as
$$
P _ {\pi} ^ {\prime} (s ^ {\prime} | s) = \sum_ {a \in A} P _ {\mathrm {e n v}} (s ^ {\prime} | s, a) \pi (a | s), \quad P _ {\pi} (s ^ {\prime}, a ^ {\prime} | s, a) = P _ {\mathrm {e n v}} (s ^ {\prime} | s, a) \pi (a ^ {\prime} | s ^ {\prime}).
$$
The stationary distribution over state-action pairs  $\mu_{\pi}$  is defined to be a nonnegative vector with coordinates summing to one and satisfying  $\mu_{\pi}^{T} = \mu_{\pi}^{T}P_{\pi}$ , while the stationary distribution over states  $\mu_{\pi}'$  is defined similarly with  $\mu_{\pi}'^T = \mu_{\pi}'^T P_{\pi}'$ . The Perron-Frobenius theorem guarantees that such a  $\mu_{\pi}$  and  $\mu_{\pi}'$  exist and are unique subject to some conditions on  $P_{\pi}', P_{\pi}$ , e.g., aperiodicity and irreducibility (Gantmacher (1964)). We use  $\mu_{\pi}(s, a)$  to denote each entry of  $\mu_{\pi}$  and  $\mu_{\pi}'(s)$  each entry of  $\mu_{\pi}'$ . Clearly,
$$
\mu_ {\pi} (s, a) = \mu_ {\pi} ^ {\prime} (s) \pi (a | s). \tag {1}
$$
The value function and the  $Q$ -function of a policy  $\pi$  is defined as:
$$
V _ {\pi} ^ {*} (s) = \sum_ {a \in A} \pi (a | s) Q _ {\pi} ^ {*} (s, a), \quad Q _ {\pi} ^ {*} (s, a) = \mathbb {E} _ {s, a, \pi} \left[ \sum_ {t = 0} ^ {+ \infty} \gamma^ {t} r \left(s _ {t}, a _ {t}\right) \right]. \tag {2}
$$
Here,  $\mathbb{E}_{s,a,\pi}$  stands for the expectation when action  $a$  is chosen in state  $s$  and all subsequent actions are chosen according to policy  $\pi$ . Throughout the paper, if  $\pi$  can be parameterized by  $\theta$ , then we will use  $\theta$  as a subscript instead of  $\pi$ , e.g., by writing  $V_{\theta}^{*}(s)$  instead of  $V_{\pi_{\theta}}^{*}(s)$ .
If  $\pi$  is parameterized by  $\theta$ , the  $Q$ -values satisfy the Bellman equation
$$
Q _ {\theta} ^ {*} (s, a) = r (s, a) + \gamma \sum_ {s ^ {\prime}, a ^ {\prime}} P _ {\theta} \left(s ^ {\prime}, a ^ {\prime} \mid s, a\right) Q _ {\theta} ^ {*} \left(s ^ {\prime}, a ^ {\prime}\right), \tag {3}
$$
which can be stated in matrix notation as
$$
Q _ {\theta} ^ {*} = R + \gamma P _ {\theta} Q _ {\theta} ^ {*}, \tag {4}
$$
where  $Q_{\theta}^{*} = (Q_{\theta}^{*}(s,a))_{(s,a)\in S\times A}$  and  $R = (R(s,a))_{(s,a)\in S\times A}$  are vectors that stack up the  $Q$ -values and rewards, respectively. We will assume rewards are bounded:
Assumption 2.1 (Bounded Reward). For any  $s, a \in S \times A$ ,  $|r(s, a)| \leq r_{\max}$ .
This assumption is commonly adopted throughout the literature, e.g., among the previous literature in Cayci et al. (2022); Wu et al. (2020). An obvious implication of this is an upper bound on the  $Q$ -values for any policy:
$$
\left| Q _ {\theta} ^ {*} (s, a) \right| \leq \frac {r _ {\operatorname* {m a x}}}{1 - \gamma}. \tag {5}
$$
2.2 The Policy Gradient Theorem
We introduce the quantity  $\phi_{\theta}(s)$ , commonly called the discounted occupation measure which is defined as
$$
\phi_ {\theta} (s) = \sum_ {t = 0} ^ {+ \infty} \gamma^ {t} P _ {\theta} (S _ {t} = s),
$$
where  $P_{\theta}(S_t = s)$  is the probability of being in state  $s$  after  $t$  steps — and recall that we always begin in state  $s_0$ . Next, we define  $\phi_{\theta}(s,a)$  as
$$
\phi_ {\theta} (s, a) = \phi_ {\theta} (s) \pi (a | s, \theta).
$$
Note that the sum of both  $\phi(s)$  and  $\phi(s, a)$  equal to  $(1 - \gamma)^{-1}$  rather than 1:
$$
\sum_ {s \in S} \phi_ {\theta} (s) = \sum_ {(s, a) \in S \times A} \phi_ {\theta} (s, a) = \frac {1}{1 - \gamma}. \tag {6}
$$
Now we are prepared to state the policy gradient theorem Sutton & Barto (2018).
Theorem 2.1. (Policy Gradient Theorem)
$$
\nabla V _ {\theta} ^ {*} = \sum_ {s \in S} \phi_ {\theta} (s) \sum_ {a \in A} Q _ {\theta} ^ {*} (s, a) \nabla \pi (a | s, \theta)
$$
It is standard to write this as
$$
\nabla V _ {\theta} ^ {*} = \sum_ {(s, a) \in S \times A} \phi_ {\theta} (s, a) Q _ {\theta} ^ {*} (s, a) \nabla \ln \pi (a | s, \theta),
$$
which can be further rewritten in matrix form as
$$
\nabla V _ {\theta} ^ {*} = \nabla \ln \pi (\theta) ^ {T} \Phi_ {\theta} Q _ {\theta} ^ {*}, \tag {7}
$$
where  $\Phi_{\theta}$  is a diagonal matrix stacking up the  $\phi_{\theta}(s,a)$  as its diagonal entries.
2.3 Parameterized Value Function and Policy
We will now state the various assumptions we have on the policies and their parametrizations. We will say that a function  $f: \mathbb{R} \to \mathbb{R}$  is  $L$ -Lipschitz if
$$
| f (x) - f (y) | \leq L | x - y |, \forall x, y,
$$
and a differentiable function  $f: \mathbb{R} \to \mathbb{R}$  is  $H$ -smooth if
$$
| \nabla f (x) - \nabla f (y) | \leq H | x - y |, \forall x, y.
$$
We will be using a multi-layer neural network to approximate the  $Q$  values under a policy. We basically follow the same setting as in Liu et al. (2020), with some changes as far as notation goes. Specifically, we define the following recursion
$$
x ^ {(k)} = \frac {1}{\sqrt {m}} \sigma \left(w ^ {(k)} x ^ {(k - 1)}\right), \text {f o r} k \in \{1, \dots , K \},
$$
where  $\sigma$  is an activation function and  $x^{(k)}$  stands for the value of  $k$ 'th layer ( $x^{(0)} \in S \times A$  is the input to this neural network). The neural network outputs  $Q(s, a, w)$ , which is defined as
$$
Q (s, a, w) = \frac {1}{\sqrt {m}} b ^ {T} x ^ {(K)}.
$$
Notice that the output is linear to  $x^{(K)}$  as no activation function is applied here. While this formulation does not have a bias, it is equivalent to a formulation with a bias if we pad all inputs with a single 1, and add an additional node to every hidden layer that propagates this 1 to subsequent layers. We will assume that all the hidden layers have the same width which we denote by  $m$ , i.e., all the matrices  $w^{(k)}$  have  $m$  rows and all the vector  $x^{(k)}, k \geq 1$  are  $m$ -dimensional. The total number of layers in the neural network is denoted by  $K$ .
For simplicity, we will make the following assumption on the neural network. Throughout the paper, we will use  $||\cdot ||$  for the standard  $l_{2}$ -norm.
Assumption 2.2. (Neural architecture and initialization) Suppose the neural network satisfies the following properties:
- (Input assumption) Any input to the neural network satisfies  $||x^{(0)}||\leq 1$
- (Activation function assumption)  $\sigma$  is  $L_{\sigma}$ -Lipschitz and  $H_{\sigma}$ -smooth.
- (Initialization assumption) Each entry of the vector  $b$  satisfies  $|b_r| \leq 1, \forall r$ , and each entry of  $w^{(k)}$  is randomly chosen from  $N(0,1)$ , independently across entries.
Liu et al. (2020) showed that with these assumptions, the following result holds with high probability - which we state as an assumption for our work.
Assumption 2.3. The absolute value of each entry of  $x^{(k)}$  (the output of layer  $k$  of the neural network) is  $\tilde{O}_m(1)$  at initialization.
Next, we will stack up the weights of different layers into a column vector  $w$  consisting of the entries of the matrices  $w^{(1)},\ldots ,w^{(K)}$ , with its norm defined by
$$
| | w | | ^ {2} = \sum_ {k = 1} ^ {K} | | w ^ {(k)} | | _ {F} ^ {2},
$$
where  $||\cdot||_F$  is the Frobenius norm. During the training process, only the weights  $w$  will be updated while the final weights  $b$  will be left to their initial value. For convenience, we define the vector  $Q(w) = (Q(s,a,w))_{(s,a)\in S\times A}$  which stacks up  $Q(s,a,w)$  over all state-action pairs  $(s,a)$ . While this vector will never be actually used in the execution of any algorithm we consider due to its high dimensionality, it will be useful in some of the arguments we will make. Finally, we assume the parametrization of the policy  $\pi$  is smooth.
Assumption 2.4 (Smooth parametrization). For all  $s, a$ , the quantities  $\pi(a|s, \theta)$ ,  $\ln \pi(a|s, \theta)$  are  $L_{\pi}$ -Lipschitz and  $L_{\pi}'$ -Lipschitz with respect to  $\theta$ , respectively.
Note that this forces us to use a smooth activation function and rules out non-differentiable activation functions such as ReLU. If a RELU-like activation is needed, one could use a GeLU or ELU activation (which are smooth versions of ReLU) and still satisfy the above assumption. Note, also, that this assumption implicitly assumes that all policies are exploratory in the sense of assigning a positive probability to each action, since the derivative of  $\ln x$  blows up as  $x\to 0$ .
2.4 Neural Actor-Critic
We will use  $\mathbf{Proj}_W\{\cdot\}$  refer to projection onto a ball with constant radius around the initial condition of the critic, where
$$
W = \{w \mid | | w - w _ {0} | | \leq \sigma_ {w} \}, \sigma_ {w} \text {i s a c o n s t a n t}.
$$
We now introduce the neural AC, which updates the actor and critic parameters as
$$
w _ {t + 1} = \mathbf {P r o j} _ {W} \left\{w _ {t} + \alpha^ {w} \delta_ {t} \nabla_ {w} Q (s _ {t}, a _ {t}, w _ {t}) \right\}, \quad \theta_ {t + 1} = \theta_ {t} - \frac {\alpha^ {\theta}}{1 - \gamma} Q (\hat {s} _ {t}, \hat {a} _ {t}, w _ {t}) \nabla_ {\theta} \ln \pi (\hat {a} _ {t} | \hat {s} _ {t}, \theta_ {t}).
$$
where  $\delta_t$  is the TD error defined by
$$
\delta_ {t} = r \left(s _ {t}, a _ {t}\right) + \gamma Q \left(s _ {t} ^ {\prime}, a _ {t} ^ {\prime}, w _ {t}\right) - Q \left(s _ {t}, a _ {t}, w _ {t}\right), \tag {8}
$$
and the samples are obtained as follows:
1. the state  $s_t$  is generated by taking a step in the Markov chain  $P_{\mathrm{env}}$  from  $s_{t-1}$ ;
2. the action  $a_{t}$  is chosen according to the policy  $\pi (a|s_t,\theta_t)$
3. the next state  $s_t'$ , i.e.,  $s_t' = s_{t+1}$ , is determined according to the transition probability  $P_{\mathrm{env}}$  of the MDP;
4. the action  $a_{t}^{\prime}$  is an action chosen at the next state according to the policy  $\pi (a|s_t^\prime ,\theta_t)$
5. the state-action pair  $(\hat{s}_t,\hat{a}_t)$  is obtained by first sampling a geometric random variable  $T$  with distribution  $\{P(T = t) = (1 - \gamma)\gamma^t,t\geq 0\}$ , and second obtaining  $T$  transitions by starting at  $s_0$  and taking actions according to  $\pi (a|s,\theta_t)$ . Note that this update has to be re-done at every step, i.e., every  $t$  requires  $\operatorname {Geom}(\gamma)$  steps.
The above algorithm will be referred to as actor-critic with Markov sampling. It is also possible to consider a simplified variant, where step 1 is slightly altered as follows: the state  $s_t$  is instead chosen i.i.d. at every step from the stationary distribution of  $\mu_{\theta_t}$  of the policy  $\pi_{\theta_t}$ . This is referred to as actor-critic with i.i.d. sampling.
2.4.1 Approximation Assumptions
It is evident that any performance bound on AC will depend on how well the neural network used for the critic can approximate the true value function. If we choose a neural network architecture for which universal approximation theorems do not apply and it happens to poorly approximate the true  $Q$ -functions, we will likely obtain poor results. Here, we will largely sidestep this issue by defining  $\epsilon$  to be the approximation quality of the critic; our final performance results will be in terms of  $\epsilon$ .
Formally, we say that the vector  $Q$  is an  $\epsilon$ -approximation to the true value function  $Q_{\theta_t}^*$  of the policy  $\pi_{\theta_t}$  if
$$
\max  _ {(s, a) \in S \times A} | Q (s, a) - Q _ {\theta_ {t}} ^ {*} (s, a) | \leq \epsilon . \tag {9}
$$
We then make the following assumption.
Assumption 2.5. (Approximation capabilities of critic) For all  $\theta$ , there exists some set of weights  $\hat{w}_{\theta}$  which give rise to an  $\epsilon$ -approximation of  $Q_{\theta}^{*}$ .
Note that, since we do not say what  $\epsilon$  is, this assumption could well be a definition of  $\epsilon$ . Throughout the paper we will use  $\hat{Q}_{\theta_t}^*$  to denote an  $\epsilon$ -approximation to  $Q_{\theta_t}^*$  guaranteed by the above assumption. Thus,
$$
Q (\hat {w} _ {\theta_ {t}}) = \hat {Q} _ {\theta_ {t}} ^ {*}.
$$
Further, we will assume that  $\hat{w}_{\theta}$  is a smooth function of  $\theta$  in the sense of its first and second derivatives.
Assumption 2.6. (Smoothness of critic approximation) Suppose there exists scalars  $L_{w}(i)$  and  $H_{w}(i)$  such that for all  $\theta$ ,
$$
| | \nabla \hat {w} _ {\theta} ^ {*} (i) | | \leq L _ {w} (i), \quad \lambda_ {\max } \{\nabla^ {2} \hat {w} _ {\theta} ^ {*} (i) \} \leq H _ {w} (i).
$$
where  $\lambda_{\mathrm{max}}\{\cdot \}$  stands for the largest eigenvalue.
For convenience, we define
$$
L _ {w} = \sqrt {\sum_ {i} L _ {w} (i) ^ {2}}, \quad H _ {w} = \sqrt {\sum_ {i} H _ {w} (i) ^ {2}}.
$$
Finally, we need an additional assumption on the critic neural network. It should be obvious that any analysis of actor-critic has to assume that the critic is capable of approximating the correct  $Q$ -values. One part of this was already assumed earlier in Assumption 2.5, where we assumed that an approximation exists. However, it should be clear that in the nonlinear case this is insufficient: just because there exists an approximation which is good doesn't follow that it will be found during training, which is not known converge to the global minimizer in the nonlinear case, but rather only to a critical point.
We thus need something to rule out the possibility that the critic training gets stuck at a bad critical point. It turns out that it suffices to assume (a quantitative version of the fact that) the critic is one-to-one map from weights to value functions.
Assumption 2.7. (State regularity) There exists some constant  $\lambda' > 0$  such that
$$
\left| \left| Q (w) - \hat {Q} _ {\theta} ^ {*} \right| \right| \geq \lambda^ {\prime} | | w - \hat {w} _ {\theta} ^ {*} | |.
$$
Let us parse the meaning of this assumption. Because  $Q(\hat{w}_{\theta}^{*}) = \hat{Q}_{\theta}^{*}$ , it is appropriately viewed as a quantitative version of the statement that if  $w_{1} \neq w_{2}$ , then  $Q(w_{1}) \neq Q(w_{2})$ . To see why this makes sense, note that the number of states is typically many magnitudes larger than the number of parameters in the critic. For example, in many applications the number of states often corresponds to the number of images (when states are captured through images) which is astronomical. Thus  $Q(w)$  will map  $w$  to a much higher dimensional space.
If the states  $s$  are generated from a probability distribution which has a continuous density, and the activation functions are continuous and increasing, the chance that  $Q_{w_1}(s) = Q_{w_2}(s)$  even for one state  $s$  is zero. That is why we label it "state regularity" as above (and recall that  $Q(w)$  stacks up  $Q_w(s)$  for every state  $s$ ).
On a technical level, this property ensures that critic actually finds a good critic approximation in spite of the nonlinearity of the update. If the features are linear, this reduces to the assumption that the features are linearly independent, an assumption which is made in all previous and related work on AC method (Wu et al. (2020); Olshevsky & Gharesifard (2022); Chen & Zhao (2022); Kumar et al. (2023)) and TD Learning (Liu & Olshevsky (2021); Xu & Gu (2020); Cai et al. (2019); Zou et al. (2019)).
2.5 The Mixing of Markov Chains
It is standard to make an assumption to the effect that all the Markov chains that can arise satisfy a mixing condition. Otherwise, it is possible under Markov sampling for the state to fail to explore the entire state-space. This assumption, first introduced by Bhandari et al. (2018) in TD learning, now is commonly used in AC analysis (Olshevsky & Gharesifard (2022); Wu et al. (2020); Chen & Zhao (2022)).