SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Feiyang Wu
feiyangwu@gatech.edu
Jingyang Ke
jingyang.ke@gatech.edu
School of Computational Science and Engineering
Atlanta, Georgia 30332
We study the problem of Inverse Reinforcement Learning (IRL) with an average-reward criterion. The goal is to recover an unknown policy and a reward function when the agent only has samples of states and actions from an experienced agent. Previous IRL methods assume that the expert is trained in a discounted environment, and the discount factor is known. This work alleviates this assumption by proposing an average-reward framework with efficient learning algorithms. We develop novel stochastic first-order methods to solve the IRL problem under the average-reward setting, which requires solving an Average-reward Markov Decision Process (AMDP) as a subproblem. To solve the subproblem, we develop a Stochastic Policy Mirror Descent (SPMD) method under general state and action spaces that needs  $\mathcal{O}(1 / \varepsilon)$  steps of gradient computation. Equipped with SPMD, we propose the Inverse Policy Mirror Descent (IPMD) method for solving the IRL problem with a  $\mathcal{O}(1 / \varepsilon^2)$  complexity. To the best of our knowledge, the aforementioned complexity results are new in IRL literature with the average reward criterion. Finally, we corroborate our analysis with numerical experiments using the MuJoCo benchmark and additional control tasks.
Reinforcement Learning (RL) problems are frequently formulated as Markov Decision Processes (MDPs). The agent learns a policy to maximize the reward gained over time. However, in numerous engineering challenges, we are typically presented with a set of state-action samples from experienced agents, or experts, without an explicit reward signal. Inverse Reinforcement Learning (IRL) aspires to recover the expert's policy and reward function from these collected samples.
Methods like Imitation Learning [29, 15, 10] reduces the IRL problem to predicting an expert's behavior without estimating the reward signal. Such formulation is undesirable in some scenarios when we either wish to continue training or analyze the structure of the agent's reward signal, e.g., in reward discovery for animal behavior study [31, 34, 26, 14]. Furthermore, as a popular methodology in Imitation Learning, Generative Adversarial Network (GAN) [15, 42] suffers from unstable training due to mode collapse [32]. The sampling-based Bayesian IRL (BIRL) approach [8, 9, 6] treats the reward function as a posterior distribution from the observation. This approach is conceptually straightforward but suffers from slow convergence of sampling. Another line of research reformulates
the IRL as RL with general utility functions [36, 39, 12, 1]. While theoretically attractive, such a formulation does not learn reward functions.
Furthermore, inferring rewards poses an additional challenge: without any additional assumption, the same expert behavior can be explained by multiple reward signals [40]. To resolve this ambiguity, the authors propose adding the principle of Maximum Entropy that favors a solution with a higher likelihood of the trajectory. In the following work [41], the objective is replaced by maximizing the entropy of the policy, which better characterizes the nature of stochastic sequential decision-making. A body of work focuses on alleviating the computational burden of the nested optimization structure of under the Maximum Entropy framework. In [11], the authors skipped the reward function update stage by representing rewards using action-value functions.  $f$ -IRL [23] minimize the divergence of state visitation distribution for finite-horizon MDPs. More recently, in [37], the authors propose a dual formulation of the original Maximum Entropy IRL and a corresponding algorithm to solve IRL in the discounted setting.
Despite the success in practice, theoretical understanding of RL/IRL methods is lacking. For RL, specifically for Discounted Markov Decision Processes (DMDPs), analysis of policy gradient methods for solving DMDPs is catching up only until recently [2][6][5][9][18]. Meanwhile, understanding of the Average-reward Markov Decision Processes (AMDPs) remains very limited, where the policy aims to maximize the long-run average reward gained. Li et al.[20] propose stochastic first-order methods for solving AMDPs with a rate of convergence  $\tilde{\mathcal{O}} (\log (\varepsilon^{-1}))$  for policy optimization, but it is constrained to finite states and actions with policy explicitly represented in tabular forms. Moreover, for IRL, the only known finite time analysis are [35, 37, 38, 21] for either solving DMDPs, or finite-horizon MDPs. There are no well-established convergence analyses for IRL with AMDPs. Moreover, prior works only focus on MDPs with finite state and action spaces, which do not fully address many real-life problems that inherently feature continuous elements (e.g., robotics [17]). Such settings further require general function approximations, the usage of which lacks theoretical understanding in recent work as well.
In summary, all previous works either do not fundamentally address our problem, which necessitates the learning of reward functions, or they miss key elements such as general state and action spaces, general function approximation, and the analysis incorporating approximation errors. Each of these facets demands substantial effort. In this paper, we focus on convergence analyses for average-award MDPs. We extend the RL algorithm [20] for AMDPs to general state and action spaces with general function approximation and develop an algorithm to solve the IRL problem with the average-reward criterion. Analyses of both algorithms are new in the literature to the best of our knowledge.
Stochastic Policy Mirror Descent for solving AMDPs: Our research introduces the Stochastic Policy Mirror Descent (SPMD) algorithm, a novel approach designed to tackle Average-reward Markov Decision Processes (AMDPs) that involve general state and action spaces. Leveraging a performance difference lemma, we demonstrate that the SPMD algorithm converges within  $\mathcal{O}(\varepsilon^{-1})$  steps for a specific nonlinear function approximation class, and in no more than  $\mathcal{O}(\varepsilon^{-2})$  steps for general function approximation.
Inverse Policy Mirror Descent for solving Maximum Entropy IRL: We expound a dual form of the Maximum Entropy IRL problem under the average-reward framework with the principle of Maximum Entropy. This dual problem explicitly targets the minimization of discrepancies between the expert's and agent's expected average reward. Consequently, we propose a first-order method named Inverse Policy Mirror Descent (IPMD) for effectively addressing the dual problem. This algorithm operates by partially resolving an Entropy Regularized RL problem at each iteration, which is systematically solved by employing SPMD. Drawing upon the two-timescale stochastic approximation analysis framework [3], we present the convergence result and establish a  $\mathcal{O}(\varepsilon^{-2})$  rate of convergence.
Numerical experiments: Our RL and IRL methodologies have been tested against the well-known robotics manipulation benchmark, MuJoCo, as a means to substantiate our theoretical analysis. The results indicate that the proposed SPMD and IPMD algorithms generally outperform state-of-the-art algorithms. In addition, we found that the IPMD algorithm notably reduces the error in recovering the reward function.
2 Background and problem setting
2.1 Average reward Markov decision processes
An Average-reward Markov decision process is described by a tuple  $\mathcal{M} \coloneqq (\mathcal{S}, \mathcal{A}, \mathsf{P}, c)$ , where  $\mathcal{S}$  denotes the state space,  $\mathcal{A}$  denotes the action space,  $\mathsf{P}$  is the transition kernel, and  $c$  is the cost function. At each time step, the agent takes action  $a \in \mathcal{A}$  at the current state  $s \in \mathcal{S}$  according to its policy  $\pi: \mathcal{S} \to \mathcal{A}$ . We use  $\Pi$  as the space of all feasible policies. Then the agent moves to the next state  $s' \in \mathcal{S}$  with probability  $\mathsf{P}(s'|s, a)$ , while the agent receives an instantaneous cost  $c(s, a)$  (or a reward  $r(s, a) = -c(s, a)$ ). The agent's goal is to determine a policy that minimizes the long-term cost
$$
\rho^ {\pi} (s) := \lim  _ {T \rightarrow \infty} \frac {1}{T} \mathbb {E} _ {\pi} \left[ \sum_ {t = 0} ^ {T - 1} \left(c \left(s _ {t}, a _ {t}\right) + h ^ {\pi} \left(s _ {t}\right)\right) \mid s _ {0} = s \right], \tag {1}
$$
where  $h^\pi$  is a closed convex function with respect to the policy  $\pi$ , i.e., there exists  $\mu_h \geq 0$  such that
$$
\left. h ^ {\pi} (s) - \left[ h ^ {\pi^ {\prime}} (s) + \left\langle \nabla h ^ {\pi^ {\prime}} (s, \cdot), \pi (\cdot | s) - \pi^ {\prime} (\cdot | s) \right\rangle \right] \geq \mu_ {h} D (\pi (s), \pi^ {\prime} (s)), \right. \tag {2}
$$
where  $\nabla h^{\pi '}$  denotes the subgradient of  $h$  at  $\pi^\prime$  and  $D(\cdot ,\cdot)$  is the Bregman distance, i.e.,
$$
D \left(a _ {2}, a _ {1}\right) := \omega \left(a _ {1}\right) - \left[ \omega \left(a _ {2}\right) + \langle \omega \left(a _ {2}\right) ^ {\prime}, a _ {1} - a _ {2} \rangle \right] \geq \frac {1}{2} \| a _ {1} - a _ {2} \| ^ {2}, \text {f o r a l l} a _ {1}, a _ {2} \in \mathcal {A}. \tag {3}
$$
Here  $\omega : \mathcal{A} \to \mathbb{R}$  is a strongly convex function with the associated norm  $\|\cdot\|$  in the action space  $\mathcal{A}$  and let us denote  $\|\cdot\|_*$  as its dual norm. In this work we also utilize a span semi-norm [27]:  $\|v\|_{sp,\infty} := \max v - \min v, \forall v \in \mathbb{R}^n$ . If  $h^\pi = 0$ , Eq.1 reduces to the classical unregularized AMDP. If  $h^\pi(s) = \mathbb{E}_\pi[-\log \pi] := \mathcal{H}(\pi(s))$ , i.e., the (differential) entropy, Eq.1 defines the average reward of the so-called entropy-regularized MDPs.
In this work, we consider the ergodic setting, for which we make the following assumption formally:
Assumption 2.1. For any feasible policy  $\pi$ , the Markov chain induced by policy  $\pi$  is ergodic. The Markov chain is Harris ergodic in general state and action spaces (see [22]). The stationary distribution  $\kappa^{\pi}$  induced by any feasible policy exists and is unique. There is some constant number  $0 < \Gamma < 1$  such that  $\kappa^{\pi}(s) \geq 1 - \Gamma$ .
As a result of Assumption 2.1, for any feasible policy  $\pi$ , the average-reward function does not depend on the initial state (see Section 8 of [27]). Given that, one can view  $\rho^{\pi}$  as a function of  $\pi$ . For a given policy we also define the basic differential value function (also called bias function; see, e.g., [27])
$$
\bar {V} ^ {\pi} (s) := \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} c \left(s _ {t}, a _ {t}\right) + h ^ {\pi} \left(s _ {t}\right) - \rho^ {\pi} \mid s _ {0} = s, a _ {t} \sim \pi (\cdot \mid s _ {t}), s _ {t + 1} \sim \mathrm {P} (\cdot \mid s _ {t}, a _ {t}) \right], \tag {4}
$$
and the basic differential action-value function (or basic differential  $Q$ -function) is defined as
$$
\bar {Q} ^ {\pi} (s, a) := \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} c \left(s _ {t}, a _ {t}\right) + h ^ {\pi} \left(s _ {t}\right) - \rho^ {\pi} \mid s _ {0} = s, a _ {0} = a, a _ {t} \sim \pi (\cdot \mid s _ {t}), s _ {t + 1} \sim \mathrm {P} (\cdot \mid s _ {t}, a _ {t}) \right]. \tag {5}
$$
Moreover, we define the sets of differential value functions and differential action-value functions (differential  $Q$ -functions) as the solution sets of the following Bellman equations, respectively,
$$
\bar {V} ^ {\pi} (s) = \mathbb {E} \left[ \bar {Q} ^ {\pi} (s, a) | a \sim \pi (\cdot | s) \right], \tag {6}
$$
$$
\bar {Q} ^ {\pi} (s, a) = c (s, a) + h ^ {\pi (s)} (s) - \rho^ {\pi} (s) + \int \mathrm {P} \left(d s ^ {\prime} \mid s, a\right) \bar {V} ^ {\pi} \left(s ^ {\prime}\right). \tag {7}
$$
Under Assumption 2.1, the solution of Eq. (6) (resp., Eq. 7) is unique up to an additive constant. Finally, our goal in solving an AMDP is to find an optimal policy  $\pi^{*}$  that minimizes the average cost:
$$
\rho^ {*} = \rho^ {\pi^ {*}} = \min  _ {\pi} \rho^ {\pi} \quad \text {s . t .} \quad \pi (\cdot | s) \in \Pi , \forall s \in \mathcal {S}. \tag {AMDP}
$$
2.2 Inverse Reinforcement Learning
Suppose there is a near-optimal policy, and we are given its demonstrations  $\zeta \coloneqq \{(s_i,a_i)\}_{i\geq 1}$ . IRL aims to recover a reward function such that the estimated reward best explains the demonstrations. We consider solving the IRL problem under the Maximum Entropy framework (MaxEnt-IRL), which
aims to find a reward representation that maximizes the entropy of the corresponding policy and incorporates feature matching as a constraint. Formally, the MaxEnt-IRL problem is described as
$$
\max  _ {\pi} \mathcal {H} (\pi) := \mathbb {E} _ {(s, a) \sim d ^ {\pi} (..)} [ - \log \pi (a | s) ] \quad \left(\operatorname {M a x E n t - I R L}\right)
$$
$$
\mathrm {s . t .} \mathbb {E} _ {(s, a) \sim d ^ {\pi}} [ \varphi (s, a) ] = \mathbb {E} _ {(s, a) \sim d ^ {E}} [ \varphi (s, a) ]
$$
where  $d^{E}$  and  $d^{\pi}$  denote the state-action distribution induced by the expert and current policy  $\pi^{E}$  and  $\pi$  respectively, and  $\varphi (s,a)\in \mathbb{R}^n$  denotes the feature of a given  $(s,a)$  pair. If we assume that for a given state-action pair, the cost is a linear function to its feature, i.e.,  $c(s,a;\theta) = \theta^T\varphi (s,a)$  for some parameter  $\theta$  with the same dimension of the feature  $\varphi (s,a)$ , we can show that the parameter  $\theta$  is the dual multiplier of the above optimization problem, as the Lagrangian function can be written as
$$
\mathcal {L} (\pi , \theta) = \mathcal {H} (\pi) + \mathbb {E} _ {(s, a) \sim d ^ {E}} [ c (s, a; \theta) ] - \mathbb {E} _ {(s, a) \sim d ^ {\pi}} [ c (s, a; \theta) ]. \tag {8}
$$
Therefore, the dual problem is formulated as
$$
\min  _ {\theta} L (\theta) := \mathbb {E} _ {(s, a) \sim d ^ {E}} [ c (s, a; \theta) ] - \mathbb {E} _ {(s, a) \sim d ^ {\pi}} [ c (s, a; \theta) ] \quad \text {(D u a l I R L)}
$$
$$
\mathrm {s . t .} \pi = \arg \max _ {\pi^ {\prime}} \mathbb {E} _ {(s, a) \sim d ^ {\pi^ {\prime}}} [ - c (s, a; \theta) ] + \mathcal {H} (\pi^ {\prime}).
$$
Notice that to find  $\pi$ , we need to solve an Entropy Regularized Reinforcement Learning problem, i.e., solving AMDP with the regularizer set to be the negative entropy,  $h^\pi = -\mathcal{H}(\pi)$ . To this end, we first propose a Stochastic Policy Mirror Descent (SPMD) method for solving AMDP. The solution methods are presented in section 3. Then we introduce an Inverse Policy Mirror Descent (IPMD) algorithm based on SPMD for the inverse RL problem (Dual IRL), introduced in section 4. We will see that we only need to solve the subproblem AMDP partially.
3 Stochastic Policy Mirror Descent for AMDPs
This section proposes an SPMD method to solve Regularized Reinforcement Learning problems for AMDPs. SPMD operates in an actor-critic fashion. In each step, the agent evaluates its current policy (critic step) and performs a policy optimization step (actor step). In this work, we assume there is a way to perform the critic step using standard methods, e.g., Temporal Difference learning using neural networks. Further discussion on implementation is included in section 3.1. We will focus on designing a novel actor step and providing its complexity analysis.
Our policy optimization algorithm is motivated by the following performance difference lemma, which characterizes the difference in objective values of two policies  $\pi, \pi'$ .
Lemma 3.1. (Performance Difference) Assume that Assumption 2.1 holds. For any  $\pi, \pi' \in \Pi$
$$
\rho^ {\pi^ {\prime}} - \rho^ {\pi} = \int \psi^ {\pi} (s, \pi^ {\prime} (s)) \kappa^ {\pi^ {\prime}} (d s), \forall s \in \mathcal {S}, \tag {9}
$$
$$
\psi^ {\pi} (s, \pi^ {\prime} (s)) := \bar {Q} ^ {\pi} (s, \pi^ {\prime} (s)) - \bar {V} ^ {\pi} (s) + h ^ {\pi^ {\prime}} (s) - h ^ {\pi} (s). \tag {10}
$$
Proof can be found in [A.1]. The above lemma shows the gradient of the objective function with respect to action  $a$  is  $\bar{Q}^{\pi}(s, \pi(s)) + h^{\pi}(s)$ , i.e.,
$$
\nabla_ {a} \rho^ {\pi} = \int \nabla_ {a} \left(\bar {Q} ^ {\pi} (s, \pi (s)) + h ^ {\pi} (s)\right) \kappa^ {\pi} (d s). \tag {11}
$$
The existence of the above equation requires the differentiability of  $\bar{Q}$ ,  $h$  and locally Lipschitz continuity of  $\kappa^{\pi}$  [see 24, 25]. Note that  $\psi$  can be seen as some generalized advantage function. Lemma 3.1 shows that the gradient of the objective relates to differential  $Q$ -functions. This inspires us to update the policy in the following mirror-descent style:
$$
\pi_ {k + 1} (s) = \underset {a \in \mathcal {A}} {\arg \min } \bar {Q} ^ {\pi_ {k}} (s, a) + h ^ {a} (s) + \frac {1}{\eta_ {k}} D \left(\pi_ {k} (s), a\right), \tag {12}
$$
where  $\eta_{k}$  is some predefined step size. The complexity analysis of this type of algorithm has been thoroughly studied in [20] in tabular forms or linear function approximation settings, i.e., a lookup
table represents the policy, and the differential  $Q$ -functions are approximated as linear functions with respect to some feature space. In practice, especially in the Deep Reinforcement Learning community, both policy and the  $Q$ -functions are represented by neural networks. Thus, novel analysis is required for general function approximation. Additionally, the exact value of  $\bar{Q}^{\pi}$  and  $\nabla \omega(\pi)$  in Eq. [19] can only be estimated through approximation. Note that  $\nabla \omega(\pi)$  rises inside the Bregman distance  $D(\pi_k(s), a)$ , referring to Eq. [13]. We consider their stochastic estimators calculated from state-action sample pairs  $\zeta$ , denoted as  $\bar{Q}^{\pi, \zeta}(s, a; \phi)$ ,  $\tilde{\nabla} \omega(\pi(s); \phi)$  respectively, where  $\phi$  denotes the parameters of the method of choice, for example, weights and biases in a neural network. For the rest of the paper, we abbreviate  $\bar{Q}^{\pi, \zeta}(s, a; \phi)$  as  $\bar{Q}(s, a; \phi)$  when the context is clear. We described the Stochastic Policy Mirror Descent (SPMD) algorithm in Algorithm [1].
Algorithm 1: The Stochastic Policy Mirror Descent (SPMD) algorithm for AMDPs
1: Input: Initialize random policy  $\pi_0$  and step size sequence  $\{\eta_k\}$
2: for  $k = 0,1,\dots ,K$  do
3: Sample collection:  $\zeta_{k} = \{(s_{t},a_{t},c_{t})\}_{t\geq 1}$
4: Critic step: Approximate  $\bar{Q}^{\pi_k}$ $\nabla \omega (\pi_k(s))$  with
$$
\bar {Q} ^ {\pi_ {k}} (s, a) \approx \bar {Q} ^ {\pi_ {k}, \zeta_ {k}} (s, a; \phi_ {k}), \tag {13}
$$
$$
\nabla \omega \left(\pi_ {k} (s)\right) \approx \tilde {\nabla} \omega \left(\pi_ {k} (s); \phi_ {k}\right). \tag {14}
$$
5: Actor step: Update the policy
$$
\pi_ {k + 1} (s) = \arg \min  _ {a \in \mathcal {A}} \bar {\mathcal {Q}} ^ {\pi_ {k}, \zeta_ {k}} (s, a; \phi_ {k}) + h ^ {a} (s) + \frac {1}{\eta_ {k}} (\langle \tilde {\nabla} \omega (\pi_ {k} (s); \phi_ {k}), a \rangle + \omega (a)). \tag {15}
$$
The following paragraphs present the complexity analysis for Algorithm [1]. Note that we assume the approximated  $Q$  function might not be convex but weakly convex, i.e.,
$$
\bar {\mathcal {Q}} (s, a; \phi) + \mu_ {\bar {\mathcal {Q}}} D \left(\pi_ {0} (s), a\right) \tag {16}
$$
is convex w.r.t.  $a \in \mathcal{A}$  for some  $\mu_{\mathcal{Q}} \geq 0$  and  $\pi_0$  is the initialized policy. This assumption is general as any differential function with Lipschitz continuous gradients is weakly convex. Moreover, we assume that  $h^{\cdot}(s)$ ,  $\bar{Q} (s,\cdot ;\theta)$  and  $\bar{Q}^{\pi}(s,\cdot)$  are Lipschitz continuous with respect to the action.
Assumption 3.2. For all  $a_1, a_2 \in \mathcal{A}$  there exist some constants  $M_h, M_{\bar{Q}}, M_{\bar{Q}}$  such that
$$
\left| h ^ {a _ {1}} (s) - h ^ {a _ {2}} (s) \right| \leq M _ {h} \| a _ {1} - a _ {2} \|, \tag {17}
$$
$$
\left| \bar {\mathcal {Q}} (s, a _ {1}; \phi) - \bar {\mathcal {Q}} (s, a _ {2}; \phi) \right| \leq M _ {\bar {\mathcal {Q}}} \| a _ {1} - a _ {2} \|, \tag {18}
$$
$$
\left| \bar {Q} ^ {\pi} (s, a _ {1}) - \bar {Q} ^ {\pi} (s, a _ {2}) \right| \leq M _ {\bar {Q}} \| a _ {1} - a _ {2} \|, \tag {19}
$$
and  $\bar{\mathcal{Q}}(s, a; \phi)$  is  $\mu_{\bar{\mathcal{Q}}}$ -weakly convex, i.e., Eq. [16] is convex.
Note that the strong convexity modulus of the objective function in Eq. [15] can be very large since  $\eta_k$  can be small, in which case the subproblem Eq. [15] is strongly convex, thus the solution of Eq. [15] is unique due to strong convexity [18].
Our complexity analysis follows the style of convex optimization. We begin the analysis by decomposing the approximation error in the following way
$$
\delta_ {k} ^ {Q} (s, a) := \bar {\mathcal {Q}} (s, a; \phi_ {k}) - \bar {Q} ^ {\pi_ {k}} (s, a), \tag {20}
$$
$$
\begin{array}{l} \delta_ {k} ^ {\omega} (s) := \frac {1}{\eta_ {k}} \tilde {\nabla} \omega (\pi_ {k} (s); \phi_ {k}) - \frac {1}{\eta_ {k}} \nabla \omega (\pi_ {k} (s)), \\ = \mathbb {E} \left[ \frac {1}{\eta_ {k}} \tilde {\nabla} \omega (\pi_ {k} (s); \phi_ {k}) \right] - \frac {1}{\eta_ {k}} \nabla \omega (\pi_ {k} (s)) + \frac {1}{\eta_ {k}} \tilde {\nabla} \omega (\pi_ {k} (s); \phi_ {k}) - \mathbb {E} \left[ \frac {1}{\eta_ {k}} \tilde {\nabla} \omega (\pi_ {k} (s); \phi_ {k}) \right]. \tag {21} \\ \end{array}
$$
We refer to the first two and last two approximation error terms in  $\delta^{\omega}$  as  $\delta_k^{\omega,det}$  and  $\delta_k^{\omega,sto}$ , respectively, as they represent a deterministic approximation error which we cannot reduce and a stochastic error related to the variance of the estimator. They represent the noise introduced in the critic step. To make
the noise tractable and the entire algorithm convergent, we must assume they are not arbitrarily large. But before making more assumptions about these error terms, we must distinguish the case when  $\bar{\mathcal{Q}} + h$  is convex and when it's not, i.e., when  $\mu \geq 0$  or  $\mu < 0$ , where  $\mu = \mu_h - \mu_{\bar{\mathcal{Q}}}$  for simplicity. We can obtain globally optimal solutions if  $\mu \geq 0$ . While in the other case, only stationary points can be obtained, e.g., both  $\psi^{\pi}(s, \pi'(s))$  and  $D(\pi_k, \pi_{k+1})$  approach 0. Note that  $\psi$  relates to the progress we make in each iteration. See [A.6] for more detail. These two cases require different assumptions on the error terms, so they must be treated differently. We start with the optimistic case when  $\mu \geq 0$ , under which we make the following assumption about the approximation error  $\delta_k^\bar{\mathcal{Q}}(s, a)$  and  $\delta_k^\omega(s)$ :
Assumption 3.3. When  $\mu \geq 0$ , the critic step has bounded errors, i.e., there exist some constants  $\varsigma^{\bar{Q}}, \varsigma^{\omega}, \sigma^{\omega}$  such that
$$
\mathbb {E} _ {\zeta_ {k}, s \sim \kappa^ {*}} \left[ \left| \delta_ {k} (s, \pi_ {k} (s)) \right| + \left| \delta_ {k} (s, \pi^ {*} (s)) \right| \right] \leq \varsigma^ {\bar {Q}}, \tag {22}
$$
$$
\mathbb {E} _ {\zeta_ {k}, s \sim \kappa^ {*}} \left[ \left\| \delta_ {k} ^ {\omega , d e t} (s) \right\| _ {*} \right] \leq \varsigma^ {\omega}, \mathbb {E} _ {\zeta_ {k}, s \sim \kappa^ {*}} \left[ \left\| \delta_ {k} ^ {\omega , s t o} \right\| _ {*} ^ {2} \right] \leq (\sigma^ {\omega}) ^ {2}, \tag {23}
$$
where  $\zeta_{k}$  is the samples we collect in each iteration.
With the above assumption, we present the complexity result of SPMD when  $\mu \geq 0$ .
Theorem 3.4. Suppose 2.1, 3.2, 3.3 hold and the step size  $\eta_{k}$  satisfies
$$
\frac {\beta_ {k}}{\eta_ {k}} \leq \beta_ {k - 1} \left(\mu + \frac {1}{\eta_ {k - 1}}\right), k \geq 1, \tag {24}
$$
for some  $\beta_{k} \geq 0$ . Then after running the SPMD algorithm for  $K$  iterations, we have