SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Once we have a bound for  $y$ , we can easily compute a bound for  $x$ .
B Actor-Critic
In this section, we will review and clarify the AC algorithm being considered in this paper.
Recall that in Eq.(8), we defined the set  $W$  as  $W = \{w\mid ||w - w_0||\leq \sigma_w\}$  and the TD error  $\delta_t$  as
$$
\delta_ {t} = r (s _ {t}, a _ {t}) + \gamma Q (s _ {t} ^ {\prime}, a _ {t} ^ {\prime}, w _ {t}) - Q (s _ {t}, a _ {t}, w _ {t}).
$$
With  $\delta_{t}$ , we now define function  $f$  and  $g$  such that
$$
f \left(O _ {t}, w _ {t}\right) = \delta_ {t} \nabla_ {w} Q \left(s _ {t}, a _ {t}, w _ {t}\right), \quad g \left(\hat {O} _ {t}, w _ {t}, \theta_ {t}\right) = Q \left(\hat {s} _ {t}, \hat {a} _ {t}, w _ {t}\right) \nabla_ {\theta} \ln \pi \left(\hat {a} _ {t} \mid \hat {s} _ {t}, \theta_ {t}\right), \tag {15}
$$
where we denote by  $O_{t} = (s_{t}, a_{t}, s_{t}', a_{t}') \in S \times A \times S \times A$  the tuple of  $s_{t}, a_{t}, s_{t}', a_{t}'$  and by  $\hat{O}_{t} = (\hat{s}_{t}, \hat{a}_{t}) \in S \times A$  the  $\hat{s}_{t}, \hat{a}_{t}$  pair. The way of sampling  $O_{t}$  and  $\hat{O}_{t}$  is mentioned in Section 2.4.
With these notations, the AC update mentioned in Section 2.4 can be written as
$$
w _ {t + 1} = \mathbf {P r o j} _ {W} \left\{w _ {t} + \alpha^ {w} f \left(O _ {t}, w _ {t}\right) \right\}, \quad \theta_ {t + 1} = \theta_ {t} - \frac {\alpha^ {\theta}}{1 - \gamma} g \left(\hat {O} _ {t}, w _ {t}, \theta_ {t}\right).
$$
We find it useful to talk about the "mean path update". This just means that the functions  $f(\cdot, \cdot)$  and  $g(\cdot, \cdot, \cdot)$  in Eq.(15) are replaced by their means, assuming that  $(s_t, a_t)$  is sampled from  $\mu_{\theta_t}$  while  $(\hat{s}_t, \hat{a}_t)$  is sampled from  $(1 - \gamma)\phi_{\theta_t}$ . More formally, the mean-path update functions  $\bar{f}(\cdot, \cdot)$  and  $\bar{g}(\cdot, \cdot)$  are defined as
$$
\begin{array}{l} \bar {f} (w _ {t}, \theta_ {t}) = \sum_ {s _ {t}, a _ {t}} \mu_ {\theta_ {t}} (s _ {t}, a _ {t}) \nabla Q (s _ {t}, a _ {t}, w _ {t}) \mathbb {E} _ {s _ {t} ^ {\prime}, a _ {t} ^ {\prime} | s _ {t}, a _ {t}} [ r (s _ {t}, a _ {t}) + \gamma Q (s _ {t} ^ {\prime}, a _ {t} ^ {\prime}, w _ {t}) - Q (s _ {t}, a _ {t}, w _ {t}) ] \\ = \nabla Q (w _ {t}) ^ {T} D _ {\theta_ {t}} (\gamma P _ {\theta_ {t}} - I) (Q (w _ {t}) - Q _ {\theta_ {t}} ^ {*}), \\ \end{array}
$$
$$
\bar {g} \left(w _ {t}, \theta_ {t}\right) = \frac {1}{1 - \gamma} \mathbb {E} _ {\hat {O} _ {t}} \left[ g \left(\hat {O} _ {t}, w _ {t}, \theta_ {t}\right) \right] = \frac {1}{1 - \gamma} \mathbb {E} \left[ g \left(\hat {O} _ {t}, w _ {t}, \theta_ {t}\right) \mid \mathcal {F} _ {t} \right] = \nabla \ln \pi \left(\theta_ {t}\right) ^ {T} \Phi_ {\theta_ {t}} Q \left(w _ {t}\right), \tag {16}
$$
where  $\mathcal{F}_t = (w_t,\theta_t)$  and  $\mathbb{E}_{O_t}$ ,  $\mathbb{E}_{\hat{O}_t}$  assume  $O_{t}$  follows  $\mu_{\theta_t}$  and  $\hat{O}_t$  follows  $(1 - \gamma)\phi_{\theta_t}$ . To show the latter one, as we discussed in Section 2.4, we first sample  $T$  such that  $P(T = t) = (1 - \gamma)\gamma^t$ . We then perform  $T$  transition starting from  $s_0$ . This means that by total probability,
$$
P (S = s) = \sum_ {t = 0} ^ {+ \infty} P (S = s | T = t) \cdot P (T = t) = (1 - \gamma) \sum_ {t = 0} ^ {+ \infty} \gamma^ {t} P (S _ {t} = s) = \phi (s).
$$
Thus, if the policy here is given by  $\theta_t$ , it follows immediately that
$$
P \left(\hat {O} _ {t} = (s, a)\right) = (1 - \gamma) \phi_ {\theta_ {t}} (s) \pi (a | s, \theta_ {t}) = (1 - \gamma) \phi_ {\theta_ {t}} (s, a).
$$
Notice that under these notations, we have  $\mathbb{E}_{\hat{O}_t}[g(\hat{O}_t,w_t,\theta_t)] = \mathbb{E}[g(\hat{O}_t,w_t,\theta_t)|\mathcal{F}_t]$ .
Algorithm 1 details the algorithm considered in this paper.
Algorithm 1 Actor-Critic
Require: Numbers of iterations  $T$ , learning rate  $\alpha^w$  and  $\alpha^\theta$ , projection set  $W$ .
Initialize  $\theta_0, b_r$  and  $w^{(k)}$  such that  $|b_r| \leq 1$ ,  $\forall r$  and every entry of  $w^{(k)}$  is chosen from  $N(0,1)$ .
Initialize the starting state-action pair  $s_0, a_0$ .
for  $t\in \{1,2,\ldots ,T\}$  do
Sample  $s_t \sim P_{\mathrm{env}}(s|s_{t-1}, a_{t-1})$ ,  $a_t \sim \pi(a|s_t, \theta_t)$ ,  $s_t' \sim P_{\mathrm{env}}(s|s_t, a_t)$ ,  $a_t' \sim \pi(a|s_t', \theta_t)$ .
Sample  $\hat{O}_t$  by first sampling a random variable  $T$  with  $P(T = t) = (1 - \gamma)\gamma^t$ , and second obtaining  $T$  transitions by starting at  $s_0$  and taking actions according to  $\pi(a|s, \theta_t)$ .
Compute  $\delta_t$ ,  $f(O_t, w_t)$ ,  $g(\hat{O}_t, w_t, \theta_t)$ , and update  $w_{t+1}$  and  $\theta_{t+1}$  as
$$
w _ {t + 1} = \operatorname {P r o j} _ {W} \left\{w _ {t} + \alpha^ {w} f \left(O _ {t}, w _ {t}\right) \right\}, \quad \theta_ {t + 1} = \theta_ {t} - \frac {\alpha^ {\theta}}{1 - \gamma} g \left(\hat {O} _ {t}, w _ {t}, \theta_ {t}\right).
$$
C Auxiliary Lemmas
In this section, we will present all the auxiliary lemmas needed to prove Theorem 3.1.
C.1 Properties of the Neural Network
In this section, we will show that the neural network has Lipschitzness and smoothness properties. The following result is based on Liu et al. (2020) and has been talked about in Tian et al. (2023).
Lemma C.1. For any  $(s,a)\in S\times A$ , there exists scalars  $L_{Q}(s,a),H_{Q}(s,a)$  such that for  $w_{1},w_{2}\in W$
$$
\left| \left| Q (s, a, w _ {1}) - Q (s, a, w _ {2}) \right| \right| \leq L _ {Q} (s, a) \left| \left| w _ {1} - w _ {2} \right| \right|.
$$
$$
\left| \left| \nabla Q (s, a, w _ {1}) - \nabla Q (s, a, w _ {2}) \right| \right| \leq H _ {Q} (s, a) \left| \left| w _ {1} - w _ {2} \right| \right|.
$$
If we further define
$$
L _ {Q} = \sqrt {\sum_ {s , a} L _ {Q} (s , a) ^ {2}}, H _ {Q} = \sqrt {\sum_ {s , a} H _ {Q} (s , a) ^ {2}},
$$
then  $L_{Q} = O(1)$  and  $H_{Q} = \tilde{O}\left(\frac{1}{\sqrt{m}}\right)$  with respect to  $m$ .
Proof. The Lipschitzness property is proved in Tian et al. (2023) while the smoothness property is a direct result of Liu et al. (2020).  $\square$
C.2 Properties of the Operator  $\mathcal{N}$
In this section, we will show several results about the operator  $\mathcal{N}_{\theta}$  defined in Eq.(12).
Lemma C.2. For any function  $f$  defined on  $S \times A$ ,
$$
- \mathcal {N} _ {\theta} (f) = f ^ {T} D _ {\theta} (\gamma P _ {\theta} - I) f.
$$
Proof. The proof is given by Lemma A.1 in Tian et al. (2023).
Lemma C.3. There exists  $\lambda_{\mathrm{min}} > 0$  and  $\lambda_{\mathrm{min}}' > 0$  such that
$$
\mathcal {N} _ {\theta} \left(Q (w) - \hat {Q} _ {\theta} ^ {*}\right) \geq \lambda_ {\min} \left\| w - \hat {w} _ {\theta} ^ {*} \right\| ^ {2},
$$
$$
\mathcal {N} _ {\theta} \left(Q (w) - \hat {Q} _ {\theta} ^ {*}\right) \geq \lambda_ {\mathrm {m i n}} ^ {\prime} \left\| Q (w) - \hat {Q} _ {\theta} ^ {*} \right\| ^ {2},
$$
where  $\lambda_{\mathrm{min}} = (1 - \gamma)\mu_{\mathrm{min}}\lambda^{\prime 2}$  and  $\lambda_{\mathrm{min}}^{\prime}$  is given by  $\lambda_{\mathrm{min}}^{\prime} = (1 - \gamma)\mu_{\mathrm{min}}\lambda^{\prime 2}L_{Q}^{2}$ .
Proof. To show the first part,
$$
\begin{array}{l} \mathcal {N} _ {\theta} \left(Q (w) - \hat {Q} _ {\theta} ^ {*}\right) = (1 - \gamma) | | Q (w) - \hat {Q} _ {\theta} ^ {*} | | _ {D} ^ {2} + \gamma | | Q (w) - \hat {Q} _ {\theta} ^ {*} | | _ {\mathrm {D i r}} ^ {2} \\ \geq (1 - \gamma) \| Q (w) - \hat {Q} _ {\theta} ^ {*} \| _ {D} ^ {2} \\ \geq (1 - \gamma) \mu_ {\min} \lambda^ {\prime 2} \| w - \hat {w} _ {\theta} ^ {*} \| ^ {2}, \\ \end{array}
$$
where the first line is the definition of  $\mathcal{N}(\cdot)$  while the last line uses Assumption 2.9. We can set  $\lambda_{\mathrm{min}} = (1 - \gamma)\mu_{\mathrm{min}}\lambda^{\prime 2}$  and we finish the proof for the first part.
The second part is an obvious result that simply combines the first part and Lemma C.1.
Lemma C.4. Suppose  $D_{Q} = \nabla \ln \pi (\theta_{t})^{T}\Phi_{\theta_{t}}\left[Q(w_{t}) - Q_{\theta_{t}}^{*}\right]$ . The relationship between  $D_{Q}$  and  $\mathcal{N}_{\theta_t}(Q(w_t) - \hat{Q}_{\theta_t}^*)$  can be described as follows:
$$
\left\| D _ {Q} \right\| ^ {2} \leq c _ {1} \cdot \mathcal {N} _ {\theta_ {t}} \left(Q (w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*}\right) + c _ {2} \epsilon^ {2},
$$
where  $c_{1} = \frac{2L^{\prime}_{\pi}^{2}}{(1 - \gamma)^{2}\lambda_{\min}^{\prime}}$  and  $c_{2} = \frac{2L^{\prime}_{\pi}^{2}}{(1 - \gamma)^{2}}$ .
Proof. One can easily show that
$$
(1 - \gamma) D _ {Q} = \mathbb {E} _ {(s, a) \sim (1 - \gamma) \phi_ {\theta_ {t}}} \left[ \ln \pi (a | s, \theta_ {t}) (Q (s, a, w _ {t}) - Q _ {\theta_ {t}} ^ {*} (s, a)) \right].
$$
As assumed in Assumption 2.4,  $||\nabla \ln \pi (a|s,\theta_t)||\leq L_\pi^\prime$  . Hence,
$$
(1 - \gamma) | | D _ {Q} | | \leq L _ {\pi} ^ {\prime} \cdot \mathbb {E} _ {(s, a) \sim (1 - \gamma) \phi_ {\theta_ {t}}} \left[ | Q (s, a, w _ {t}) - Q _ {\theta_ {t}} ^ {*} (s, a) | \right].
$$
Using the facts that  $(\mathbb{E}[X])^2 \leq \mathbb{E}[X^2]$ ,
$$
(1 - \gamma) ^ {2} | | D _ {Q} | | ^ {2} \leq L _ {\pi} ^ {\prime 2} \cdot \mathbb {E} _ {(s, a) \sim (1 - \gamma) \phi_ {\theta_ {t}}} \left[ | Q (s, a, w _ {t}) - Q _ {\theta_ {t}} ^ {*} (s, a) | ^ {2} \right].
$$
$$
| Q (s, a, w _ {t}) - Q _ {\theta_ {t}} ^ {*} (s, a) | ^ {2} \leq 2 | Q (s, a, w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*} (s, a) | ^ {2} + 2 \epsilon^ {2}.
$$
where we use Assumption 2.5 which tells us  $|\hat{Q}_{\theta_t}^*(s, a) - Q_{\theta_t}^*(s, a)| \leq \epsilon$ . Hence,
$$
(1 - \gamma) ^ {2} | | D _ {Q} | | ^ {2} \leq 2 L _ {\pi} ^ {\prime} ^ {2} \cdot \mathbb {E} _ {(s, a) \sim (1 - \gamma) \phi_ {\theta_ {t}}} \left[ | Q (s, a, w _ {t}) - \hat {Q} _ {\theta_ {t}} ^ {*} (s, a) | ^ {2} \right] + 2 L _ {\pi} ^ {\prime} ^ {2} \epsilon^ {2}.
$$
Combine with Lemma C.3 and the fact that  $(1 - \gamma)\phi_{\theta_t}(s,a)\leq 1$
$$
(1 - \gamma) ^ {2} | | D _ {Q} | | ^ {2} \leq \frac {2 L _ {\pi} ^ {\prime 2}}{\lambda_ {\mathrm {m i n}} ^ {\prime}} \mathcal {N} _ {\theta} \left(Q (w) - \hat {Q} _ {\theta} ^ {*}\right) + 2 L _ {\pi} ^ {\prime 2} \epsilon^ {2}.
$$
C.3 Mean-value Theorem and Extensions
Lemma C.5. These following lemmas generalize the mean-value theorem to higher dimensional input and output cases.
(a) Let  $h: \mathbb{R} \to \mathbb{R}$  be any differentiable function. For any  $x, y \in \mathbb{R}$ , there exists  $\lambda \in (0,1)$  and  $z = \lambda x + (1 - \lambda)y$  such that
$$
h (y) - h (x) = h ^ {\prime} (z) (y - x).
$$
(b) Let  $\xi : \mathbb{R}^a \to \mathbb{R}$  be any differentiable function. For any  $x, y \in \mathbb{R}^a$ , there exists  $\lambda \in (0,1)$  and  $z = \lambda x + (1 - \lambda)y$  such that
$$
\xi (y) - \xi (x) = \xi^ {\prime} (z) (y - x).
$$
(c) Let  $f: \mathbb{R}^a \to \mathbb{R}^b$  be any differentiable function and  $e \in \mathbb{R}^b$  be any vector. For any  $x, y \in \mathbb{R}^a$ , there exists  $\lambda \in (0,1)$  and  $z = \lambda x + (1 - \lambda)y$  such that
$$
e ^ {T} (f (y) - f (x)) = e ^ {T} f ^ {\prime} (z) (y - x),
$$
where  $f'(z)$  is the Jacobian at  $z$ .
Proof. This proof is given by Lemma A.2 in Tian et al. (2023).
C.4 The Mixing of Two Markov Chains
In this section, we argue that if two Markov chains satisfy Assumption 2.8, then the difference between their distributions could be very small. This is inspired by and follows the same logic as Chen & Zhao (2022); Zou et al. (2019). Before that, we will first introduce the total variation norm for vectors and matrices, which can be used to measure a difference between distributions.
Denote  $f: X \to R$  to be any real value function. We can define the total variation norm of  $f$ , denoted by  $||f||_{\mathrm{TV}}$ , as
$$
| | f | | _ {\mathrm {T V}} = \sum_ {x \in X} | f (x) |.
$$
For matrix  $A$ , we can also define  $\| A \|_{\mathrm{TV}}$  to be
$$
\| A \| _ {\mathrm {T V}} = \sup  _ {| | f | | _ {\mathrm {T V}} = 1} \| f ^ {T} A \| _ {\mathrm {T V}}.
$$
If  $f$  is some probability measure, since  $f(x) \in [0,1]$ , it is easy to conclude that  $||f||_{\mathrm{TV}} = 1$ . Likewise, if  $A$  is a Markov transition matrix, we can show that  $||A||_{\mathrm{TV}} = 1$ .
The following lemma establishes the relationship between the total variation norm with the more familiar 1-norm and  $\infty$ -norm.
Lemma C.6. The following statements are true:
a. For any vector  $f$ ,  $||f||_{TV} = ||f||_1$ .
b. For any matrix  $A$ ,  $\| A \|_{TV} = \| A \|_{\infty}$ .
Proof. The lemma is obvious so we omit the proof here.
Based on Assumption 2.8, we have the following result:
Lemma C.7. If the Markov chain has transition probability matrix  $A$ , then we have
$$
\left| \left| A ^ {t} \right| \right| _ {\mathrm {T V}} \leq 1, \forall t.
$$
Further, if the Markov chain satisfies Assumption 2.8, then we have
$$
\left| \left| A ^ {t} \right| \right| _ {\mathrm {T V}} \leq C \beta^ {t}, \forall t \geq \tau_ {\mathrm {m i x}}.
$$
Proof. The first part of this lemma is obvious because  $A$  is a stochastic (Markov) matrix and by Lemma C.6,  $||\cdot||_{\mathrm{TV}}$  is just the same as  $||\cdot||_{\infty}$ .
For the second part, by the definition of total variation norm,
$$
| | A | | _ {\mathrm {T V}} = \sup  _ {| | f | | _ {T V} = 1} | | f ^ {T} A | | _ {\mathrm {T V}} = \frac {1}{2} \sup  _ {i, j} | | (e _ {i} - e _ {j}) ^ {T} A | | _ {\mathrm {T V}},
$$
where  $e_i$  means the all-zero vector except a 1 at the  $i$ 'th entry. By Assumption 2.8,
$$
\left| \left| A ^ {t} \right| \right| _ {\mathrm {T V}} \leq \frac {1}{2} \sup  _ {i, j} \left(\left| \left| \left(e _ {i} - \mu^ {\prime}\right) ^ {T} A ^ {t} \right| \right| _ {\mathrm {T V}} + \left| \left| \left(e _ {j} - \mu^ {\prime}\right) ^ {T} A ^ {t} \right| \right| _ {\mathrm {T V}}\right) \leq C \beta^ {t}, \forall t \geq \tau_ {\operatorname {m i x}}.
$$
The following theorem, which is inspired by Theorem 3.1 in Mitrophanov (2005), is very important in many analyses of AC that take Markov sampling into consideration (i.e., Wu et al. (2020); Chen & Zhao (2022)). However, since our settings are slightly different, we provide our own version.
Lemma C.8. Suppose we have the following two Markov Chains which of both satisfy Assumption 2.8,
$$
p _ {0} ^ {A} \stackrel {{A}} {{\longrightarrow}} p _ {1} ^ {A} \stackrel {{A}} {{\longrightarrow}} \ldots \stackrel {{A}} {{\longrightarrow}} p _ {t} ^ {A},
$$
$$
p _ {0} ^ {B} \stackrel {{B}} {{\longrightarrow}} p _ {1} ^ {B} \stackrel {{B}} {{\longrightarrow}} \dots \stackrel {{B}} {{\longrightarrow}} p _ {t} ^ {B},
$$
where  $p_i^A, p_i^B$  stand for the probability at step  $i$  under transition matrix  $A, B$ , respectively. The following inequality holds:
$$
| | p _ {t} ^ {A} - p _ {t} ^ {B} | | _ {1} \leq C \beta^ {t} | | p _ {0} ^ {A} - p _ {0} ^ {B} | | _ {1} + (\tau_ {\mathrm {m i x}} + C \frac {1}{1 - \beta}) | | A - B | | _ {\infty}, \forall t \geq \tau_ {\mathrm {m i x}} + 1.
$$
Proof. First, we prove that
$$
p _ {t} ^ {A ^ {T}} - p _ {t} ^ {B ^ {T}} = \left(p _ {0} ^ {A} - p _ {0} ^ {B}\right) ^ {T} B ^ {t} + \sum_ {i = 0} ^ {t - 1} p _ {0} ^ {A ^ {T}} (A - B) B ^ {t - i - 1}.
$$
by induction. If  $t = 1$ , by definition we know
$$
p _ {1} ^ {A ^ {T}} - p _ {1} ^ {B ^ {T}} = p _ {0} ^ {A ^ {T}} A - p _ {0} ^ {B ^ {T}} B = \left(p _ {0} ^ {A} - p _ {0} ^ {B}\right) ^ {T} B + p _ {0} ^ {A ^ {T}} (A - B).
$$
If the result holds for  $t = k$ , then when  $t = k + 1$
$$
\begin{array}{l} p _ {k + 1} ^ {A} ^ {T} - p _ {k + 1} ^ {B} ^ {T} = p _ {k} ^ {A T} A - p _ {k} ^ {B T} B \\ = \left(p _ {k} ^ {A} - p _ {k} ^ {B}\right) ^ {T} B + p _ {k} ^ {A T} (A - B) \\ = \left(p _ {0} ^ {A} - p _ {0} ^ {B}\right) ^ {T} B ^ {k + 1} + \sum_ {i = 0} ^ {k - 1} p _ {0} ^ {A T} (A - B) B ^ {k - i} + p _ {k} ^ {A T} (A - B) \\ = \left(p _ {0} ^ {A} - p _ {0} ^ {B}\right) ^ {T} B ^ {k + 1} + \sum_ {i = 0} ^ {k} p _ {0} ^ {A T} (A - B) B ^ {k - i}, \\ \end{array}
$$
where the third line is because we assume the result holds for the  $t = k$  case. Now, we can take the total variation norm on both sides:
$$
\begin{array}{l} \left\| p _ {t} ^ {A} - p _ {t} ^ {B} \right\| _ {T V} \leq \left\| \left(p _ {0} ^ {A} - p _ {0} ^ {B}\right) ^ {T} B ^ {t} \right\| _ {T V} + \left\| \sum_ {i = 0} ^ {t - 1} p _ {0} ^ {A T} (A - B) B ^ {t - i - 1} \right\| _ {T V} \\ \leq \left\| p _ {0} ^ {A} - p _ {0} ^ {B} \right\| _ {T V} \cdot \left\| B ^ {t} \right\| _ {T V} + \left\| p _ {0} ^ {A} \right\| _ {T V} \cdot \| A - B \| _ {T V} \cdot \sum_ {i = 0} ^ {t - 1} \left\| B ^ {t - i - 1} \right\| _ {T V} \\ \leq C \beta^ {t} \left\| p _ {0} ^ {A} - p _ {0} ^ {B} \right\| _ {1} + \| A - B \| _ {\infty} \left(\tau_ {\mathrm {m i x}} + \sum_ {i = \tau_ {\mathrm {m i x}}} ^ {t - 1} C \beta^ {i}\right) \\ = C \beta^ {t} \left\| p _ {0} ^ {A} - p _ {0} ^ {B} \right\| _ {1} + \| A - B \| _ {\infty} \left(\tau_ {\mathrm {m i x}} + C \frac {\beta^ {\tau_ {\mathrm {m i x}}} - \beta^ {t}}{1 - \beta}\right) \\ \leq C \beta^ {t} | | p _ {0} ^ {A} - p _ {0} ^ {B} | | _ {1} + (\tau_ {\mathrm {m i x}} + C \frac {1}{1 - \beta}) | | A - B | | _ {\infty}, \\ \end{array}
$$
where the third line utilizes Lemma C.6 and Lemma C.7.
With Lemma C.8, we can derive many useful results. The following result is similar to Lemma 3 in Zou et al. (2019), Lemma B.1 in Wu et al. (2020), and Lemma B.4 in Chen & Zhao (2022), which shows that both of  $\mu_{\theta}^{\prime}$ , the stationary distribution over states, and  $\mu_{\theta}$ , the stationary distribution over state-action pairs, are Lipschitz with respect to  $\theta$ .
Lemma C.9. The following statements hold:
$$
| | \mu_ {\theta_ {1}} ^ {\prime} - \mu_ {\theta_ {2}} ^ {\prime} | | _ {1} \leq (\tau_ {\mathrm {m i x}} + C \frac {1}{1 - \beta}) n _ {a} L _ {\pi} | | \theta_ {1} - \theta_ {2} | |.
$$
$$
\left| \left| \mu_ {\theta_ {1}} - \mu_ {\theta_ {2}} \right| \right| _ {1} \leq \left(1 + \tau_ {\operatorname {m i x}} + C \frac {1}{1 - \beta}\right) n _ {a} L _ {\pi} \left| \left| \theta_ {1} - \theta_ {2} \right| \right|,
$$
where  $n_a$  is the number of actions. In other words,  $n_a = |A|$ .
Proof. Recall the state transition matrix  $P_{\theta}^{\prime}$  is defined by  $P_{\theta}(s'|s) = \sum_{a} P_{\mathrm{env}}(s'|s, a)\pi(a|s, \theta)$  and corresponding stationary distribution  $\mu_{\theta}^{\prime T} = \mu_{\theta}^{\prime T}P_{\theta}^{\prime}$ . Use Lemma C.9, we know that
$$
| | \mu_ {\theta_ {1}} ^ {\prime} - \mu_ {\theta_ {2}} ^ {\prime} | | _ {1} \leq C \beta^ {t} | | \mu_ {\theta_ {1}} ^ {\prime} - \mu_ {\theta_ {2}} ^ {\prime} | | _ {1} + (\tau_ {\text {m i x}} + C \frac {1}{1 - \beta}) | | P _ {\theta_ {1}} ^ {\prime} - P _ {\theta_ {2}} ^ {\prime} | | _ {\infty}.
$$
$$
\begin{array}{l} | | P _ {\theta_ {1}} ^ {\prime} - P _ {\theta_ {2}} ^ {\prime} | | _ {\infty} = \sup _ {s} \sum_ {s ^ {\prime}} \sum_ {a} P _ {\mathrm {e n v}} (s ^ {\prime} | s, a) | \pi (a | s, \theta_ {1}) - \pi (a | s, \theta_ {2}) | \\ \leq \sup  _ {s} \sum_ {s ^ {\prime}} \sum_ {a} P _ {\text {e n v}} \left(s ^ {\prime} | s, a\right) L _ {\pi} | | \theta_ {1} - \theta_ {2} | | \\ = n _ {a} L _ {\pi} \left\| \theta_ {1} - \theta_ {2} \right\|, \\ \end{array}
$$
where we use Assumption 2.4 in the second line. Hence,
$$
| | \mu_ {\theta_ {1}} ^ {\prime} - \mu_ {\theta_ {2}} ^ {\prime} | | _ {1} \leq C \beta^ {t} | | \mu_ {\theta_ {1}} ^ {\prime} - \mu_ {\theta_ {2}} ^ {\prime} | | _ {1} + (\tau_ {\mathrm {m i x}} + C \frac {1}{1 - \beta}) n _ {a} L _ {\pi} | | \theta_ {1} - \theta_ {2} | |.
$$
Taking  $\lim_{t\to \infty}$  on both sides, we derive
This finishes the proof of the first part.
Now, Eq.(1) implies
$$
\begin{array}{l} \left| \left| \mu_ {\theta_ {1}} - \mu_ {\theta_ {2}} \right| \right| _ {1} = \sum_ {s, a} \left| \mu_ {\theta_ {1}} ^ {\prime} (s) \pi (a | s, \theta_ {1}) - \mu_ {\theta_ {2}} ^ {\prime} (s) \pi (a | s, \theta_ {2}) \right| \\ \leq \sum_ {s, a} \left[ | \mu_ {\theta_ {1}} ^ {\prime} (s) \pi (a | s, \theta_ {1}) - \mu_ {\theta_ {1}} ^ {\prime} (s) \pi (a | s, \theta_ {2}) | + | \mu_ {\theta_ {1}} ^ {\prime} (s) \pi (a | s, \theta_ {2}) - \mu_ {\theta_ {2}} ^ {\prime} (s) \pi (a | s, \theta_ {2}) | \right] \\ \leq \sum_ {s, a} \mu_ {\theta_ {1}} ^ {\prime} (s) L _ {\pi} | | \theta_ {1} - \theta_ {2} | | + | | \mu_ {\theta_ {1}} ^ {\prime} - \mu_ {\theta_ {2}} ^ {\prime} | | _ {1} \\ \leq \left(1 + \tau_ {\mathrm {m i x}} + C \frac {1}{1 - \beta}\right) n _ {a} L _ {\pi} | | \theta_ {1} - \theta_ {2} | |. \\ \end{array}
$$
In order to show the following lemmas, we need to introduce the following auxiliary Markov chain:
$$
\begin{array}{l} s _ {t - \tau_ {\mathrm {m i x}} - 1} \xrightarrow {\pi (\theta_ {t - \tau_ {\mathrm {m i x}} - 1})} a _ {t - \tau_ {\mathrm {m i x}} - 1} \xrightarrow {P _ {\mathrm {e n v}}} s _ {t - \tau_ {\mathrm {m i x}}} \xrightarrow {\pi (\theta_ {t - \tau_ {\mathrm {m i x}}})} a _ {t - \tau_ {\mathrm {m i x}}} \xrightarrow {P _ {\mathrm {e n v}}} s _ {t - \tau_ {\mathrm {m i x}} + 1} \xrightarrow {\pi (\theta_ {t - \tau_ {\mathrm {m i x}}})} \\ \tilde {a} _ {t - \tau_ {\mathrm {m i x}} + 1} \xrightarrow {P _ {\mathrm {e n v}}} \tilde {s} _ {t - \tau_ {\mathrm {m i x}} + 2} \xrightarrow {\pi (\theta_ {t - \tau_ {\mathrm {m i x}}})} \tilde {a} _ {t - \tau_ {\mathrm {m i x}} + 2} \xrightarrow {P _ {\mathrm {e n v}}} \dots \xrightarrow {P _ {\mathrm {e n v}}} \tilde {s} _ {t} \xrightarrow {\pi (\theta_ {t - \tau_ {\mathrm {m i x}}})} \tilde {a} _ {t} \xrightarrow {P _ {\mathrm {e n v}}} \tilde {s} _ {t + 1}. \\ \end{array}
$$
For reference, the original Markov chain around  $t$  is
$$
\begin{array}{l} s _ {t - \tau_ {\mathrm {m i x}} - 1} \xrightarrow {\pi (\theta_ {t - \tau_ {\mathrm {m i x}} - 1})} a _ {t - \tau_ {\mathrm {m i x}} - 1} \xrightarrow {P _ {\mathrm {e n v}}} s _ {t - \tau_ {\mathrm {m i x}}} \xrightarrow {\pi (\theta_ {t - \tau_ {\mathrm {m i x}}})} a _ {t - \tau_ {\mathrm {m i x}}} \xrightarrow {P _ {\mathrm {e n v}}} s _ {t - \tau_ {\mathrm {m i x}} + 1} \xrightarrow {\pi (\theta_ {t - \tau_ {\mathrm {m i x}} + 1})} \\ a _ {t - \tau_ {\mathrm {m i x}} + 1} \xrightarrow {P _ {\mathrm {e n v}}} s _ {t - \tau_ {\mathrm {m i x}} + 2} \xrightarrow {\pi (\theta_ {t - \tau_ {\mathrm {m i x}} + 2})} a _ {t - \tau_ {\mathrm {m i x}} + 2} \xrightarrow {P _ {\mathrm {e n v}}} \dots \xrightarrow {P _ {\mathrm {e n v}}} s _ {t} \xrightarrow {\pi (\theta_ {t})} a _ {t} \xrightarrow {P _ {\mathrm {e n v}}} s _ {t + 1}. \\ \end{array}
$$
For the consistency of notations, we will denote  $O_{\tau} = (s_{\tau}, a_{\tau}, s_{\tau}'', a_{\tau}')$  and  $\tilde{O}_{\tau} = (\tilde{s}_{\tau}, \tilde{a}_{\tau}, \tilde{s}_{\tau}', \tilde{a}_{\tau}')$  where in this case we have  $s_{\tau}' = s_{\tau + 1}$ ,  $\tilde{s}_{\tau}' = \tilde{s}_{\tau + 1}$  and  $a_{\tau}' \sim \pi(a|s, \theta_{\tau})$ ,  $\tilde{a}_{\tau}' \sim \pi(a|s, \theta_{t - \tau_{\mathrm{mix}}})$ . This kind of notations will immediately imply  $P(\tilde{O}_{t - \tau_{\mathrm{mix}} - 1} \in \cdot) = P(O_{t - \tau_{\mathrm{mix}} - 1} \in \cdot)$ .
The following lemma claims that the distribution difference between the two Markov chains above will be very small.
Lemma C.10. The following statements are true:
a. For any possible  $\tau \in \{t - \tau_{\mathrm{mix}}, t - \tau_{\mathrm{mix}} + 1, \dots, t\}$ ,
$$
\left| \left| P \left(S _ {\tau + 1} \in \cdot\right) - P \left(\tilde {S} _ {\tau + 1} \in \cdot\right) \right| \right| _ {1} \leq \left| \left| P \left(O _ {\tau} \in \cdot\right) - P \left(\tilde {O} _ {\tau} \in \cdot\right) \right| \right| _ {1}.
$$
b. For any possible  $\tau \in \{t - \tau_{\mathrm{mix}}, t - \tau_{\mathrm{mix}} + 1, \dots, t\}$ ,
$$
\left| \left| P \left(O _ {\tau} \in \cdot\right) - P \left(\tilde {O} _ {\tau} \in \cdot\right) \right| \right| _ {1} \leq 2 n _ {a} L _ {\pi} \mathbb {E} \left[ \left| \left| \theta_ {\tau} - \theta_ {t - \tau_ {\operatorname* {m i x}}} \right| \right| \right] + \left| \left| P \left(S _ {\tau} \in \cdot\right) - P \left(\tilde {S} _ {\tau} \in \cdot\right) \right| \right| _ {1}.
$$
c. Consider  $P(O_{t}\in \cdot)$  and  $P(\tilde{O}_t\in \cdot)$