SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \Xi \left(O _ {t}, \omega_ {t}, \boldsymbol {\theta} _ {t}\right) \right] = \mathbb {E} \left[ \Xi \left(O _ {t}, \omega_ {t}, \boldsymbol {\theta} _ {t}\right) - \Xi \left(O _ {t}, \omega_ {t}, \boldsymbol {\theta} _ {t - \tau}\right) \right] \\ + \mathbb {E} [ \Xi (O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t - \tau}) - \Xi (O _ {t}, \boldsymbol {\omega} _ {t - \tau}, \boldsymbol {\theta} _ {t - \tau}) ] \\ + \mathbb {E} [ \Xi (O _ {t}, \boldsymbol {\omega} _ {t - \tau}, \boldsymbol {\theta} _ {t - \tau}) - \Xi (\widetilde {O} _ {t}, \boldsymbol {\omega} _ {t - \tau}, \boldsymbol {\theta} _ {t - \tau}) ] \\ + \mathbb {E} [ \Xi (\widetilde {\boldsymbol {O}} _ {t}, \boldsymbol {\omega} _ {t - \tau}, \boldsymbol {\theta} _ {t - \tau}) ] \\ \leq C _ {2} \left\| \boldsymbol {\theta} _ {t} - \boldsymbol {\theta} _ {t - \tau} \right\| + 2 U _ {\delta} B \left\| \boldsymbol {\omega} _ {t} - \boldsymbol {\omega} _ {t - \tau} \right\| \\ + 2 U _ {\delta} ^ {2} B | \mathcal {A} | L _ {\pi} G \tau (\tau + 1) \alpha_ {t - \tau} + 4 U _ {\delta} ^ {2} B m \rho^ {\tau - 1}. \\ \end{array}
$$
Proof of Lemma C.6.
Proof. We will divide the proof of this lemma into three steps.
$$
\left| \Theta \left(O _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) - \Theta \left(\widetilde {O} _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) \right| \leq \left(2 U _ {\delta} B L _ {J ^ {\prime}} + 3 L _ {J} L _ {h}\right) \| \boldsymbol {\theta} _ {t} - \boldsymbol {\theta} _ {t - \tau} \|, \tag {44}
$$
where  $L_{h} = U_{\delta}L_{l} + (2 + 2\lambda^{-2} + 3\lambda^{-1})BU_{r}|\mathcal{A}|L_{\pi}(1 + \lceil \log_{\rho}m^{-1}\rceil +1 / (1 - \rho))$
Since  $\Theta (O,\pmb {\theta}) = \langle \nabla J(\pmb {\theta}),\mathbb{E}_{\pmb{\theta}}[h(O^{\prime},\pmb {\theta})] - h(O,\pmb {\theta})\rangle$  , we will show that each term in  $\Theta (O,\pmb {\theta})$  is Lipschitz.
For the term  $\nabla J(\pmb {\theta})$  , by Lemma B.3 we know it's  $L_{J^{\prime}}$  -Lipschitz.
For term  $h(O, \pmb{\theta})$ , denote  $\delta(O, \pmb{\theta}) := r(s, a) - r(\pmb{\theta}) + (\phi(s') - \phi(s))^\top \pmb{\omega}^*$ , we have
$$
\begin{array}{l} \left\| h (O, \boldsymbol {\theta} _ {1}) - h (O, \boldsymbol {\theta} _ {2}) \right\| \\ = \| \delta (O, \boldsymbol {\theta} _ {1}) \nabla \log \pi_ {\boldsymbol {\theta} _ {1}} (a | s) - \delta (O _ {t}, \boldsymbol {\theta} _ {2}) \nabla \log \pi_ {\boldsymbol {\theta} _ {2}} (a | s) \| \\ \leq \left\| \delta (O, \boldsymbol {\theta} _ {1}) \nabla \log \pi_ {\boldsymbol {\theta} _ {1}} (a | s) - \delta (O, \boldsymbol {\theta} _ {1}) \nabla \log \pi_ {\boldsymbol {\theta} _ {2}} (a | s) \right\| \\ + \| \delta (O, \boldsymbol {\theta} _ {1}) \nabla \log \pi_ {\boldsymbol {\theta} _ {2}} (a | s) - \delta (O, \boldsymbol {\theta} _ {2}) \nabla \log \pi_ {\boldsymbol {\theta} _ {2}} (a | s) \| \\ \leq U _ {\delta} L _ {l} \left\| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \right\| + B | \delta (O, \boldsymbol {\theta} _ {1}) - \delta (O, \boldsymbol {\theta} _ {2}) | \\ \leq U _ {\delta} L _ {l} \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \| + B (| r (\boldsymbol {\theta} _ {1}) - r (\boldsymbol {\theta} _ {2}) | + \| \phi (s ^ {\prime}) - \phi (s) \| \cdot \| \boldsymbol {\omega} ^ {*} (\boldsymbol {\theta} _ {1}) - \boldsymbol {\omega} ^ {*} (\boldsymbol {\theta} _ {2}) \|) \\ \leq \left(U _ {\delta} L _ {l} + 2 B L _ {*}\right) \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \| + B | \mathbb {E} _ {s \sim \mu_ {\boldsymbol {\theta} _ {1}}, a \sim \pi_ {\boldsymbol {\theta} _ {1}}} [ r (s, a) ] - \mathbb {E} _ {s \sim \mu_ {\boldsymbol {\theta} _ {1}}, a \sim \pi_ {\boldsymbol {\theta} _ {2}}} [ r (s, a) ] | \\ \leq \left(U _ {\delta} L _ {l} + 2 B L _ {*}\right) \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \| + 2 B U _ {r} d _ {T V} \left(\mu_ {\boldsymbol {\theta} _ {1}} \otimes \pi_ {\boldsymbol {\theta} _ {1}}, \mu_ {\boldsymbol {\theta} _ {2}} \otimes \pi_ {\boldsymbol {\theta} _ {2}}\right) \\ \leq \left(U _ {\delta} L _ {l} + 2 B L _ {*} + 2 B U _ {r} | \mathcal {A} | L _ {\pi} \left(1 + \lceil \log_ {\rho} m ^ {- 1} \rceil + \frac {1}{1 - \rho}\right)\right) \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \|. \\ \end{array}
$$
Hence we have  $h(O, \pmb{\theta})$  is  $L_{h}$ -Lipschitz, where  $L_{h}$  denotes the above coefficient.
$$
\begin{array}{l} \left\| \mathbb {E} _ {\boldsymbol {\theta} _ {1}} \left[ h \left(O ^ {\prime}, \boldsymbol {\theta} _ {1}\right) \right] - \mathbb {E} _ {\boldsymbol {\theta} _ {2}} \left[ h \left(O ^ {\prime}, \boldsymbol {\theta} _ {2}\right) \right] \right\| \\ \leq \left\| \mathbb {E} _ {\boldsymbol {\theta} _ {1}} \left[ h \left(O ^ {\prime}, \boldsymbol {\theta} _ {1}\right) \right] - \mathbb {E} _ {\boldsymbol {\theta} _ {1}} \left[ h \left(O ^ {\prime}, \boldsymbol {\theta} _ {2}\right) \right] \right\| + \left\| \mathbb {E} _ {\boldsymbol {\theta} _ {1}} \left[ h \left(O ^ {\prime}, \boldsymbol {\theta} _ {2}\right) \right] - \mathbb {E} _ {\boldsymbol {\theta} _ {2}} \left[ h \left(O ^ {\prime}, \boldsymbol {\theta} _ {2}\right) \right] \right\| \\ \leq \mathbb {E} _ {\boldsymbol {\theta} _ {1}} \left[ \| h \left(O ^ {\prime}, \boldsymbol {\theta} _ {1}\right) - h \left(O ^ {\prime}, \boldsymbol {\theta} _ {2}\right) \| \right] + \| \mathbb {E} _ {\boldsymbol {\theta} _ {1}} \left[ h \left(O ^ {\prime}, \boldsymbol {\theta} _ {2}\right) \right] - \mathbb {E} _ {\boldsymbol {\theta} _ {2}} \left[ h \left(O ^ {\prime}, \boldsymbol {\theta} _ {2}\right) \right] \| \\ \leq L _ {h} \left\| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \right\| + \left\| \mathbb {E} _ {\boldsymbol {\theta} _ {1}} \left[ h \left(O ^ {\prime}, \boldsymbol {\theta} _ {2}\right) \right] - \mathbb {E} _ {\boldsymbol {\theta} _ {2}} \left[ h \left(O ^ {\prime}, \boldsymbol {\theta} _ {2}\right) \right] \right\| \\ \leq L _ {h} \left\| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \right\| + 2 B U _ {r} d _ {T V} \left(\mu_ {\boldsymbol {\theta} _ {1}} \otimes \pi_ {\boldsymbol {\theta} _ {1}}, \mu_ {\boldsymbol {\theta} _ {2}} \otimes \pi_ {\boldsymbol {\theta} _ {2}}\right) \\ \leq \left[ L _ {h} + 2 B U _ {r} | \mathcal {A} | L _ {\pi} \left(1 + \lceil \log_ {\rho} m ^ {- 1} \rceil + \frac {1}{1 - \rho}\right) \right] \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \| \\ \leq 2 L _ {h} \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \|. \\ \end{array}
$$
Then we have  $\nabla J(\pmb{\theta})$  is  $L_{J}$ -bounded and  $L_{J'}$ -Lipschitz;  $h(O, \pmb{\theta}) - \mathbb{E}_{\pmb{\theta}}[h(O', \pmb{\theta})]$  is  $3L_{h}$ -Lipschitz and  $2U_{\delta}B$ -bounded. By the triangle inequality, we have
$$
\Theta \left(O _ {t}, \boldsymbol {\theta} _ {t}\right) - \Theta \left(O _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) \leq \left(2 U _ {\delta} B L _ {J ^ {\prime}} + 3 L _ {J} L _ {h}\right) \| \boldsymbol {\theta} _ {t} - \boldsymbol {\theta} _ {t - \tau} \|
$$
Step 2: show that for  $t \geq \tau > 0$ , we have
$$
\left| \mathbb {E} \left[ \Theta \left(O _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) - \Theta \left(\widetilde {O} _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) \right] \right| \leq 2 U _ {\delta} B L _ {J} | \mathcal {A} | L _ {\pi} \sum_ {k = t - \tau} ^ {t} \| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} _ {t - \tau} \| \tag {45}
$$
By definition of  $\Theta (O,\theta)$ , we have
$$
\begin{array}{l} \left. \left| \mathbb {E} \left[ \Theta \left(O _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) - \Theta \left(\widetilde {O} _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) \right] \right| \right. \\ = | \mathbb {E} [ \langle \nabla J (\boldsymbol {\theta} _ {t - \tau}), h (\widetilde {O} _ {t}, \boldsymbol {\theta} _ {t - \tau}) - h (O _ {t}, \boldsymbol {\theta} _ {t - \tau}) ] | \\ = | \mathbb {E} [ \langle \nabla J (\boldsymbol {\theta} _ {t - \tau}), h (\widetilde {O} _ {t}, \boldsymbol {\theta} _ {t - \tau}) \rangle - \mathbb {E} [ \langle \nabla J (\boldsymbol {\theta} _ {t - \tau}), h (O _ {t}, \boldsymbol {\theta} _ {t - \tau}) \rangle ] | \\ \leq 4 U _ {\delta} B L _ {J} d _ {T V} \left(\mathbb {P} \left(O _ {t} \in \cdot | s _ {t - \tau + 1}, \boldsymbol {\theta} _ {t - \tau}\right), \mathbb {P} \left(\widetilde {O} _ {t} \in \cdot | s _ {t - \tau + 1}, \boldsymbol {\theta} _ {t - \tau}\right)\right), \tag {46} \\ \end{array}
$$
where the inequality comes from the definition of total variation distance. The total variation norm between  $O_{t}$  and  $\widetilde{O}_{t}$  has been computed in (33). Plugging (33) into (46), we get
$$
\left| \mathbb {E} \left[ \Theta \left(O _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) - \Theta \left(\widetilde {O} _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) \right] \right| \leq 2 U _ {\delta} B L _ {J} | \mathcal {A} | L _ {\pi} \sum_ {k = t - \tau} ^ {t} \| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} _ {t - \tau} \|.
$$
Step 3: show that for  $t \geq \tau > 0$ , we have
$$
\left. \left| \mathbb {E} \left[ \Theta \left(\widetilde {O} _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) - \Theta \left(O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t - \tau}\right) \right] \right| \leq 4 U _ {\delta} B L _ {J} m \rho^ {\tau - 1}. \right. \tag {47}
$$
From the definition of  $\Theta (O,\pmb {\theta})$  , we have
$$
\begin{array}{l} \left. \right.\left| \mathbb {E} \left[ \Theta \left(\widetilde {O} _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) - \Theta \left(O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t - \tau}\right)\right]\right| = \left| \mathbb {E} \left[ \langle \nabla J (\boldsymbol {\theta} _ {t - \tau}), h \left(O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t - \tau}\right) \rangle - \langle \nabla J (\boldsymbol {\theta} _ {t - \tau}), h \left(\widetilde {O} _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) \rangle \right]\right| \\ \leq 4 U _ {\delta} B L _ {J} d _ {T V} \left(\mathbb {P} \left(\widetilde {O} _ {t} \in \cdot | s _ {t - \tau + 1}, \boldsymbol {\theta} _ {t - \tau}\right), \mu_ {\boldsymbol {\theta} _ {t - \tau}} \otimes \pi_ {\boldsymbol {\theta} _ {t - \tau}} \otimes \mathcal {P}\right). \\ \end{array}
$$
The inequality is due to the definition of total variation distance. From Assumption 3.2, we know that
$$
\left| \mathbb {E} \left[ \Theta \left(\tilde {O} _ {t}, \boldsymbol {\theta} _ {t - \tau} - \Theta \left(O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t - \tau}\right) \right] \right| \leq 4 U _ {\delta} B L _ {J} m \rho^ {\tau - 1}. \right.
$$
Combining (44), (45), and (47), we can decompose the Markovian bias as
$$
\begin{array}{l} \mathbb {E} \left[ \Theta \left(O _ {t}, \boldsymbol {\theta} _ {t}\right) \right] = \mathbb {E} \left[ \Theta \left(O _ {t}, \boldsymbol {\theta} _ {t}\right) - \Theta \left(O _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) \right] \\ + \mathbb {E} \left[ \Theta \left(O _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) - \Theta \left(\widetilde {O} _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) \right] \\ + \mathbb {E} \left[ \Theta \left(\tilde {\boldsymbol {O}} _ {t}, \boldsymbol {\theta} _ {t - \tau}\right) - \Theta \left(O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t - \tau}\right) \right] \\ + \mathbb {E} \left[ \Theta \left(O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t - \tau}\right) \right], \\ \end{array}
$$
where  $\widetilde{O}_t$  is from the auxiliary Markovian chain defined in (10) and  $O_t^\prime$  is from the stationary distribution which satisfies  $\mathbb{E}[\Theta (O_t',\pmb{\theta}_{t - \tau})] = 0$ .
$$
\begin{array}{l} \mathbb {E} [ \Theta (O _ {t}, \boldsymbol {\theta} _ {t}) ] \leq (2 U _ {\delta} B L _ {J ^ {\prime}} + 3 L _ {J} L _ {h}) \mathbb {E} \| \boldsymbol {\theta} _ {t} - \boldsymbol {\theta} _ {t - \tau} \| \\ + 2 U _ {\delta} B L _ {J} | \mathcal {A} | L _ {\pi} \sum_ {k = t - \tau} ^ {t} \| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} _ {t - \tau} \| + 4 U _ {\delta} B L _ {J} m \rho^ {\tau - 1} \\ \leq \left(2 U _ {\delta} B L _ {J ^ {\prime}} + 3 L _ {J} L _ {h}\right) \sum_ {k = t - \tau + 1} ^ {t} \mathbb {E} \| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} _ {k - 1} \| \\ + 2 U _ {\delta} B L _ {J} \left| \mathcal {A} \right| L _ {\pi} \sum_ {k = t - \tau + 1} ^ {t} \sum_ {j = t - \tau + 1} ^ {k} \mathbb {E} \left\| \boldsymbol {\theta} _ {j} - \boldsymbol {\theta} _ {j - 1} \right\| + 4 U _ {\delta} B L _ {J} m \rho^ {\tau - 1} \\ \leq \left(2 U _ {\delta} B L _ {J ^ {\prime}} + 3 L _ {J} L _ {h}\right) \sum_ {k = t - \tau + 1} ^ {t} \mathbb {E} \| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} _ {k - 1} \| \\ + 2 U _ {\delta} B L _ {J} | \mathcal {A} | L _ {\pi} \tau \sum_ {j = t - \tau + 1} ^ {t} \mathbb {E} \| \boldsymbol {\theta} _ {j} - \boldsymbol {\theta} _ {j - 1} \| + 4 U _ {\delta} B L _ {J} m \rho^ {\tau - 1} \\ \leq D _ {1} (\tau + 1) \sum_ {k = t - \tau + 1} ^ {t} \mathbb {E} \| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} _ {k - 1} \| + D _ {2} m \rho^ {\tau - 1} \\ \leq D _ {1} (\tau + 1) ^ {2} G \alpha + D _ {2} m \rho^ {\tau - 1} \\ \end{array}
$$
where  $D_{1} = \max \{U_{\delta}BL_{J^{\prime}} + 3L_{J}L_{h},2U_{\delta}BL_{J}|\mathcal{A}|L_{\pi}\}$  and  $D_{2} = 4U_{\delta}BL_{J}$ . Thus we conclude the proof.
E IID Sampling Analysis
Algorithm 2 Single-timescale Actor-Critic (i.i.d. sampling)
1: Input initial actor parameter  $\theta_0$ , initial critic parameter  $\omega_0$ , initial reward estimator  $\eta_0$ , stepsize  $\alpha_t$  for actor,  $\beta_t$  for critic and  $\gamma_t$  for reward estimator.
2: for  $t = 0,1,2,\dots ,T - 1$  do
3: Sample  $s_t \sim \mu_{\pmb{\theta}_t}$
4: Take the action  $a_{t}\sim \pi_{\pmb{\theta}_{t}}(\cdot |s_{t})$
5: Observe next state  $s_t' \sim \mathcal{P}(\cdot | s_t, a_t)$  and the reward  $r_t = r(s_t, a_t)$ .
6:  $\delta_t = r_t - \eta_t + \phi (s_t')^\top \pmb {\omega}_t - \phi (s_t)^\top \pmb {\omega}_t$
7:  $\eta_{t + 1} = \eta_t + \gamma_t(r_t - \eta_t)$
8:  $\omega_{t + 1} = \Pi_{U_{\omega}}(\omega_t + \beta_t\delta_t\phi (s_t))$
9:  $\pmb{\theta}_{t + 1} = \pmb{\theta}_t + \alpha_t\delta_t\nabla_\pmb{\theta}\log \pi_{\pmb{\theta}_t}(a_t|s_t)$
10: end for
Note that under i.i.d. sampling in Algorithm 2, we denote by  $s_t$  the samples from the stationary distribution and  $s_t'$  the subsequent state following transition kernel  $s_t' \sim \mathcal{P}(\cdot | s_t, a_t)$ . Correspondingly, we redefine the observation tuple as  $O_t = (s_t, a_t, s_t')$  (in contrast to  $O_t = (s_t, a_t, s_{t+1})$  in the Markovian sampling case). This modification implies the decoupling of  $O_t$  and  $O_{t+1}$  since  $s_{t+1}$  in tuple  $O_{t+1}$  is a new state sampled from the stationary distribution rather than inherited from  $O_t$ . This intuitively elucidates the vanishment of Markovian noise under i.i.d. sampling.
Lemma E.1. Under i.i.d sampling, we have
$$
\mathbb {E} \left[ \Phi \left(O _ {t}, \eta_ {t}, \boldsymbol {\theta} _ {t}\right) \right] = 0,
$$
$$
\mathbb {E} \left[ \Psi \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \right] = 0,
$$
$$
\mathbb {E} \left[ \Theta \left(O _ {t}, O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t}\right) \right] = 0,
$$
$$
\mathbb {E} \left[ \Xi \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \right] = 0.
$$
Proof. Note that the expectation is taken over all the random variables. We use the notation  $O_{t}$  to denote the tuple  $(s_t, a_t, s_t')$  and  $v_{0:t}$  to denote the sequence  $(s_t, a_t, s_t'), (s_t, a_t, s_t'), \dots, (s_t, a_t, s_t')$ . By definition in (13), it can be shown that
$$
\begin{array}{l} \mathbb {E} \left[ \Phi \left(O _ {t}, \eta_ {t}, \boldsymbol {\theta} _ {t}\right) \right] = \mathbb {E} _ {v _ {0: t}} \left[ \Phi \left(O _ {t}, \eta_ {t}, \boldsymbol {\theta} _ {t}\right) \right] \\ = \mathbb {E} _ {v _ {0: t - 1}} \mathbb {E} _ {v _ {0: t}} [ (\eta_ {t} - J (\boldsymbol {\theta} _ {t})) (r _ {t} - J (\boldsymbol {\theta} _ {t})) | v _ {0: t - 1} ], \\ \end{array}
$$
where is second equality is due to law of total expectation. Once we know  $v_{0:t-1}$ ,  $\eta_t$  and  $J(\pmb{\theta}_t)$  is not a random variable any more. It holds that
$$
\begin{array}{l} \mathbb {E} \left[ \Phi \left(O _ {t}, \eta_ {t}, \boldsymbol {\theta} _ {t}\right) \right] = \mathbb {E} _ {v _ {0: t - 1}} \mathbb {E} _ {v _ {0: t}} \left[ \left(\eta_ {t} - J (\boldsymbol {\theta} _ {t})\right) \left(r _ {t} - J (\boldsymbol {\theta} _ {t})\right) \mid v _ {0: t - 1} \right] \\ = \mathbb {E} _ {v _ {0: t - 1}} \left(\eta_ {t} - J \left(\boldsymbol {\theta} _ {t}\right)\right) \mathbb {E} _ {v _ {0: t}} \left[ \left(r _ {t} - J \left(\boldsymbol {\theta} _ {t}\right)\right) | v _ {0: t - 1} \right] \\ = \mathbb {E} _ {v _ {0: t - 1}} \left(\eta_ {t} - J \left(\boldsymbol {\theta} _ {t}\right)\right) \mathbb {E} _ {O _ {t}} \left[ \left(r _ {t} - J \left(\boldsymbol {\theta} _ {t}\right)\right) \mid v _ {0: t - 1} \right] \\ = 0, \\ \end{array}
$$
where the last equation is due to  $\mathbb{E}_{O_t}[(r_t - J(\pmb{\theta}_t))|v_{0:t-1}] = 0$  under i.i.d. sampling.
By a similar argument, we have
$$
\begin{array}{l} \mathbb {E} \left[ \Psi \left(O _ {t}, \eta_ {t}, \boldsymbol {\theta} _ {t}\right) \right] = \mathbb {E} _ {v _ {0: t}} \left[ \left\langle \boldsymbol {\omega} _ {t} - \boldsymbol {\omega} _ {t} ^ {*}, g (O, \boldsymbol {\omega}, \boldsymbol {\theta}) - \bar {g} \left(\boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \right\rangle \right] \\ = \mathbb {E} _ {v _ {0: t - 1}} \mathbb {E} _ {v _ {0: t}} [ \left\langle \boldsymbol {\omega} _ {t} - \boldsymbol {\omega} _ {t} ^ {*}, g (O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}) - \bar {g} (\boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}) \right\rangle | v _ {0: t - 1} ] \\ = \mathbb {E} _ {v _ {0: t - 1}} \left\langle \boldsymbol {\omega} _ {t} - \boldsymbol {\omega} _ {t} ^ {*}, \mathbb {E} _ {v _ {0: t}} \left[ g \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) - \bar {g} \left(\boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \right\rangle \mid v _ {0: t - 1} \right] \\ = \mathbb {E} _ {v _ {0: t - 1}} \left\langle \boldsymbol {\omega} _ {t} - \boldsymbol {\omega} _ {t} ^ {*}, \mathbb {E} _ {O _ {t}} \left[ g \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) - \bar {g} \left(\boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \right\rangle \mid v _ {0: t - 1} \right] \\ = 0, \\ \end{array}
$$
where we use the fact that  $\mathbb{E}_{O_t}[g(O_t,\pmb {\omega}_t,\pmb {\theta}_t) - \bar{g} (\pmb {\omega}_t,\pmb {\theta}_t)\rangle |v_{0:t - 1}] = 0$
$$
\begin{array}{l} \mathbb {E} \left[ \Theta \left(O _ {t}, O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t}\right) \right] = \mathbb {E} _ {v _ {0: t}} \left[ \left\langle \nabla J (\boldsymbol {\theta} _ {t}), \mathbb {E} _ {O _ {t} ^ {\prime}} \left[ h \left(O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t}\right) \right] - h \left(O _ {t}, \boldsymbol {\theta} _ {t}\right) \right\rangle \right] \\ = \mathbb {E} _ {v _ {0: t - 1}} \mathbb {E} _ {v _ {0: t}} [ \langle \nabla J (\boldsymbol {\theta} _ {t}), \mathbb {E} _ {O _ {t} ^ {\prime}} [ h (O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t}) ] - h (O _ {t}, \boldsymbol {\theta} _ {t}) \rangle | v _ {0: t - 1} ] \\ = \mathbb {E} _ {v _ {0: t - 1}} \left\langle \nabla J (\boldsymbol {\theta} _ {t}), \mathbb {E} _ {v _ {0: t}} \left[ \mathbb {E} _ {O _ {t} ^ {\prime}} [ h (O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t}) ] - h (O _ {t}, \boldsymbol {\theta} _ {t}) \right\rangle | v _ {0: t - 1} \right] \\ = \mathbb {E} _ {v _ {0: t - 1}} \left\langle \nabla J (\boldsymbol {\theta} _ {t}), \mathbb {E} _ {O _ {t}} \left[ \mathbb {E} _ {O _ {t} ^ {\prime}} \left[ h \left(O _ {t} ^ {\prime}, \boldsymbol {\theta} _ {t}\right) \right] - h (O _ {t}, \boldsymbol {\theta} _ {t}) \right\rangle | v _ {0: t - 1} \right\rangle \\ = 0, \\ \end{array}
$$
where we use fact that  $O_{t} = O_{t}^{\prime}$  under i.i.d. sampling.
The proof of  $\mathbb{E}[\Xi (O_t,\pmb {\omega}_t,\pmb {\theta}_t)] = 0$  is the same as the above argument, which concludes the proof.
Proof of Theorem 3.6.
Proof. The proof follows similarly to the Markovian sampling case. Specifically, all the Markovian noises (see the definitions in (13)) present in the former analysis reduce to zero after taking expectations. The detailed results and proof are presented in Lemma E.1. Then, replacing Lemma C.1, Lemma C.3, and Lemma C.6 with Lemma E.1, we will get the desired  $\mathcal{O}(T^{-\frac{1}{2}})$  convergence rate and thus an  $\mathcal{O}(\epsilon^{-2})$  sample complexity accordingly.