SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {T} ^ {(T)} V ^ {(T)} = r _ {\pi} + \gamma \mathcal {P} _ {\pi} V ^ {(P)} - V ^ {(P)} + \gamma \mathcal {P} _ {\pi} V ^ {(T)}.
$$
For contraction proof, consider two vectors  $u, v \in \mathbb{R}^{|\mathcal{S}|}$ .
$$
\begin{array}{l} \left\| \mathcal {T} ^ {(T)} u - \mathcal {T} ^ {(T)} v \right\| _ {\infty} = \left\| r _ {\pi} + \gamma \mathcal {P} _ {\pi} V ^ {(P)} - V ^ {(P)} + \gamma \mathcal {P} _ {\pi} u - (r _ {\pi} + \gamma \mathcal {P} _ {\pi} V ^ {(P)} - V ^ {(P)} + \gamma \mathcal {P} _ {\pi} v) \right\| _ {\infty}, \\ = \| \gamma \mathcal {P} _ {\pi} (u - v) \| _ {\infty}, \\ \end{array}
$$
$$
\boxed {\leq \gamma \| u - v \| _ {\infty}.}
$$
Therefore, the transient value function Bellman operator is a contraction mapping with contraction factor  $\gamma$ .
Theorem B.1.3. The unique fixed point of the transient operator is  $(I - \gamma \mathcal{P}_{\pi})^{-1}r_{\pi} - V^{(P)}$
Proof. We get the fixed point,  $V_{\pi}^{(T)}$ , by repeatedly applying the operator to a vector  $v$ :
$$
\begin{array}{l} V _ {\pi} ^ {(T)} = r _ {\pi} + \gamma \mathcal {P} _ {\pi} V ^ {(P)} - V ^ {(P)} + \gamma \mathcal {P} _ {\pi} v, \\ = r _ {\pi} + \gamma \mathcal {P} _ {\pi} V ^ {(P)} - V ^ {(P)} + \gamma \mathcal {P} _ {\pi} \left(r _ {\pi} + \gamma \mathcal {P} _ {\pi} V ^ {(P)} - V ^ {(P)} + \gamma \mathcal {P} _ {\pi} v\right), \\ = r _ {\pi} + \gamma \mathcal {P} _ {\pi} r _ {\pi} + (\gamma \mathcal {P} _ {\pi}) ^ {2} V ^ {(P)} - V ^ {(P)} + (\gamma \mathcal {P} _ {\pi}) ^ {2} v, \\ = r _ {\pi} + \gamma \mathcal {P} _ {\pi} r _ {\pi} + (\gamma \mathcal {P} _ {\pi}) ^ {2} V ^ {(P)} + \dots + (\gamma \mathcal {P} _ {\pi}) ^ {n} V ^ {(P)} - V ^ {(P)} + (\gamma \mathcal {P} _ {\pi}) ^ {n} v, \\ \boxed {= (I - \gamma \mathcal {P} _ {\pi}) ^ {- 1} r _ {\pi} - V ^ {(P)}.} \\ \end{array}
$$
We applied the operator to  $v$  in steps 1 to 4 and used the property  $\lim_{n\to \infty}(\gamma \mathcal{P}_\pi)^n\to 0$  in the penultimate step as  $\gamma \mathcal{P}_{\pi}$  is a sub-stochastic matrix. Many terms cancel out as it is a telescoping sum.
Theorem B.1.4. Let  $\mathcal{M}$  be an MDP in which the agent is performing transient value function updates for  $\pi$ . Define an MRP,  $\widetilde{\mathcal{M}}$ , with state space  $\widetilde{\mathcal{S}}: \{(S_t, A_t, S_{t+1}), \forall t\}$ , transition dynamics  $\widetilde{\mathcal{P}}(x'|x, a') = \mathcal{P}(s''|s', a')$ ,  $\forall x, x' \in \widetilde{\mathcal{S}}, \forall s', s'' \in \mathcal{S}, \forall a' \in \mathcal{A}$  and reward function  $\widetilde{\mathcal{R}}(x, a') = \mathcal{R}(s, a) + \gamma V^{(P)}(s') - V^{(P)}(s)$ . Then, the fixed point of Eq.(5) is the value function of  $\pi$  in  $\widetilde{\mathcal{M}}$ .
Proof. The state space of the new MDP  $\widetilde{\mathcal{M}}$  is made up of all possible consecutive tuples of  $(S_t, A_t, S_{t+1})$  pairs. The rewards in this MDP are defined as  $\widetilde{\mathcal{R}}(x, a') = \mathcal{R}(s, a) + \gamma V^{(P)}(s') - V^{(P)}(s), \forall x \in \widetilde{S}, \forall a' \in \mathcal{A}$ . The transition matrix  $\widetilde{\mathcal{P}}(x'|x, a') = \text{prob}(x'|x, a') = \text{prob}(s', a', s''|s, a, s', a') = \text{prob}(s''|s', a') = \mathcal{P}(s''|s', a')$  using Markov property of  $\mathcal{M}$ . Then, the value of state  $x = (s, a, s')$  is:
$$
v _ {\pi} (x) = v _ {\pi} \left(s, a, s ^ {\prime}\right) = \mathbb {E} _ {\tau} \left[ Z _ {t} \mid X _ {t} = \left(s, a, s ^ {\prime}\right) \right],
$$
where  $Z_{t} = \sum_{k = t}^{\infty}\gamma^{k - t}\tilde{R}_{k + 1}$  is the returns starting from state  $X_{t} = (s,a,s^{\prime})$ . Without loss of generality, consider a sample trajectory generated according to  $\pi$ ,  $\tau_{k} = (S_{0} = s,A_{0} = a,R_{1},S_{1} = s^{\prime},A_{1},R_{2},S_{2},A_{2},R_{3},\ldots ,R_{T},S_{T})$  where  $S_{T}$  is the terminal state. The returns  $Z^{\tau_k}$  of this trajectory is:
$$
\begin{array}{l} Z ^ {\tau_ {k}} = \sum_ {k = 0} ^ {\infty} \gamma^ {k} \tilde {R} _ {k + 1} \\ = \sum_ {k = 0} ^ {\infty} \gamma^ {k} \left(R _ {k + 1} + \gamma V ^ {(P)} \left(S _ {k + 1}\right) - V ^ {(P)} \left(S _ {k}\right)\right), \\ = \sum_ {k = 0} ^ {\infty} \gamma^ {k} R _ {k + 1} - V ^ {(P)} (S _ {0} = s), \\ \boxed {= G ^ {\tau_ {k}} - V ^ {(P)} (s).} \\ \end{array}
$$
Returns of every trajectory end up being the difference between the returns in the original MDP,  $G^{\tau_k}$  and the permanent value function. Most importantly, these returns only depend on the state  $s$  and it is the same for all  $a$  and  $s'$  choices following  $s$ . The value of state  $x = (s, a, s')$  can then be simplified using this observation:
$$
\begin{array}{l} v _ {\pi} (s, a, s ^ {\prime}) = \mathbb {E} _ {\tau} \left[ Z _ {t} \mid X _ {t} = (s, a, s ^ {\prime}) \right], \\ = \mathbb {E} _ {\tau} \left[ G _ {t} - V ^ {(P)} (s) \mid S _ {t} = s, A _ {t} = \cdot , S _ {t + 1} = \cdot \right], \\ \boxed {= V _ {\pi} ^ {(T)} (s).} \\ \end{array}
$$
B.2 Permanent Value Function Results
We focus on permanent value function in the next two theorems. First, we characterize the fixed point of the permanent value function. We use assumption 1 on the task distribution for the following proofs:
Theorem B.2.1. Following Theorem 2, under assumption 1 and Robbins-Monro step-size conditions, the sequence of updates computed by Eq.(4) contracts to a unique fixed point  $\mathbb{E}_{\tau}[v_{\tau}]$ .
Proof. The permanent value function is updated using Eq. (4) and samples stored in the buffer  $\mathcal{B}$  at the end of each task. This update rule can be simplified as:
$$
\begin{array}{l} V _ {k + 1} ^ {(P)} (s) \gets V _ {k} ^ {(P)} (s) + \overline {{\alpha}} _ {k} \sum_ {i = 1} ^ {L} \mathbb {I} _ {(s _ {i} = s)} (V _ {\tau} ^ {(P T)} (s) - V _ {k} ^ {(P)} (s)), \\ = V _ {k} ^ {(P)} (s) + \overline {{\alpha}} _ {k} \bigl (V _ {\tau} ^ {(P T)} (s) - V _ {k} ^ {(P)} (s) \bigr) \sum_ {i = 1} ^ {L} \mathbb {I} _ {(s _ {i} = s)}, \\ = V _ {k} ^ {(P)} (s) + \bar {\alpha} _ {k} L (s) \left(V _ {\tau} ^ {(P T)} (s) - V _ {k} ^ {(P)} (s)\right), \\ = V _ {k} ^ {(P)} (s) + \widetilde {\alpha} _ {k} (s) \left(V _ {\tau} ^ {(P T)} (s) - V _ {k} ^ {(P)} (s)\right), \\ \end{array}
$$
where  $L$  is the length of the buffer,  $\mathbb{I}$  is the indicator function,  $L(s)$  is the number of samples for state  $s$  in the buffer  $\mathcal{B}$ ,  $\widetilde{\alpha}_k(s) = \overline{\alpha}_k L(s)$ . The above update is performed once for every state in  $\mathcal{B}$  at the end of each task. We assume that the buffer has enough samples for each state so that all states are updated. This can be ensured by storing all the samples in  $\mathcal{B}$  for a long but finite period of time. The updates performed to permanent value function until that timestep will not affect the fixed point since we are in the tabular setting.
Let the sequence of learning rate  $\widetilde{\alpha}_k(s)$  obey the conditions:  $\sum_{n=1}^{\infty} \widetilde{\alpha}_n(s) = \infty$  and  $\sum_{n=1}^{\infty} \widetilde{\alpha}_n^2(s) = 0$ .
We can rewrite the PM update rule by separating the noise term:
$$
V _ {k + 1} ^ {(P)} (s) \leftarrow V _ {k} ^ {(P)} (s) + \widetilde {\alpha} _ {k} (s) \left(V _ {\tau} ^ {(P T)} (s) - V _ {k} ^ {(P)} (s)\right),
$$
$$
\begin{array}{l} = V _ {k} ^ {(P)} (s) + \widetilde {\alpha} _ {k} (s) (\mathbb {E} \left[ V _ {\tau} ^ {(P T)} (s) \right] - V _ {k} ^ {(P)} (s) + V _ {\tau} ^ {(P T)} (s) - \mathbb {E} \left[ V _ {\tau} ^ {(P T)} (s) \right]), \\ = V _ {k} ^ {(P)} (s) + \widetilde {\alpha} _ {k} (s) (\mathbb {E} \left[ V _ {\tau} ^ {(P T)} (s) \right] - V _ {k} ^ {(P)} (s) + w _ {k}). \\ \end{array}
$$
Let  $\mathcal{F}_k$  denote the history of the algorithm. Then,  $\mathbb{E}[w_k(s)|\mathcal{F}_k] = \mathbb{E}[V_\tau^{(PT)}(s) - \mathbb{E}[V_\tau^{(PT)}(s)]] = 0$  and under the assumption  $\mathbb{E}[w_k^2 (s)|\mathcal{F}_k]\leq A + BV_k^{(P)^2}(s)$  (finite noise variance), we conclude that the permanent value function converges to  $E[V_{\tau}^{(PT)}(s)]$ . But  $E[V_{\tau}^{(PT)}(s)]$  is contracting towards  $\mathbb{E}[v_{\tau}(s)]$ , therefore, the permanent value function is contracting towards  $\mathbb{E}[v_{\tau}(s)]$ .
Theorem B.2.2. The fixed point of the permanent value function optimizes the jumpstart objective,  $J = \arg \min_{u\in \mathbb{R}^{|S|}}\frac{1}{2}\mathbb{E}_{\tau}[\| u - v_{\tau}\| _2^2 ]$ .
Proof. The jumpstart objective function is given by  $J = \arg \min_{u\in \mathbb{R}^{|S|}}\frac{1}{2}\mathbb{E}_{\tau}[\| u - v_{\tau}\| _2^2 ]$ . Let  $K = \frac{1}{2}\mathbb{E}_{\tau}[\| u - v_{\tau}\| _2^2]$ . We get the optimal point by differentiating  $K$  with respect to  $u$  and then equating it to 0:
$$
\begin{array}{l} \nabla_ {u} K = \nabla_ {u} \left(\frac {1}{2} \mathbb {E} _ {\tau} [ \| u - v _ {\tau} \| _ {2} ^ {2} ]\right), \\ = \frac {1}{2} \mathbb {E} _ {\tau} \left[ \nabla_ {u} \| u - v _ {\tau} \| _ {2} ^ {2} \right], \\ = \frac {1}{2} \mathbb {E} _ {\tau} \left[ 2 \left(u - v _ {\tau}\right) \right], \\ = u - \mathbb {E} _ {\tau} \left[ v _ {\tau} \right], \\ 0 = u - \mathbb {E} _ {\tau} [ v _ {\tau} ], \\ \boxed {u = \mathbb {E} _ {\tau} [ v _ {\tau} ].} \\ \end{array}
$$
Therefore, the fixed point of the permanent value function also optimizes the jumpstart objective.
B.3 Semi-Continual RL Results
We assume that at time  $t$ , both  $V^{(TD)}$  and  $V^{(PT)}$  have converged to the true value function of task  $i$  and the agent gets samples from task  $\tau$  from timestep  $t + 1$  onward for the following theorems.
Theorem B.3.1. Under assumption 1, there exists  $k_{0}$  such that  $\forall k\geq k_0,\mathbb{E}_{\tau}[\left\| V^{(TD)} - v_i\right\| _2^2 ] > \mathbb{E}_{\tau}[\left\| V^{(P)} - v_i\right\| _2^2 ],\forall i.$
Proof. Let  $\left\| V^{(P)} - v_{i}\right\|_{2}^{2} = D_{i}$ , which is a constant because we are updating transient value function only. Let  $\left\| v_{\tau} - v_{i}\right\|_{2}^{2} = \Delta_{\tau,i}$  denote the distance between value functions of two tasks  $i$  and  $\tau$ . Clearly,  $\Delta_{\tau,i} = \Delta_{i,\tau} \neq 0$  and  $\Delta_{i,i} = 0$ . And, let  $Err_{t}^{i,\tau} = \left\| V_{t}^{(TD)} - v_{i}\right\|_{2}^{2}$  denote prediction error with respect to the previous task  $i$  at timestep  $t$  for TD estimates.
At timestep  $t$ ,  $V^{(PT)} = V^{(TD)} = v_{i}$ . The agent starts getting samples from task  $\tau$  from timestep  $t + 1$ . Using the result from Bhandari et al. [6]:
$$
\begin{array}{l} \left\| V _ {t + 1} ^ {(T D)} - v _ {\tau} \right\| _ {2} ^ {2} \leq \exp \left\{\frac {- \omega (1 - \gamma^ {2})}{4} \right\} \left\| V _ {t} ^ {(T D)} - v _ {\tau} \right\| _ {2} ^ {2}, \\ = \exp \left\{\frac {- (1 - \gamma^ {2})}{4} \right\} \Delta_ {\tau , i}, \\ \end{array}
$$
since the smallest eigenvalue of the feature matrix  $\omega = 1$  for tabular approximations. Let  $C = \exp \left\{\frac{-(1 - \gamma^2)}{4}\right\}$ . Now we can apply the above result recursively for  $k$  timesteps to get
$$
\left\| V _ {t + k} ^ {(T D)} - v _ {\tau} \right\| _ {2} ^ {2} \leq C ^ {k} \Delta_ {\tau , i}.
$$
Then the error with respect to the past task after  $k$  timesteps is,
$$
E r r _ {t + k} ^ {i, \tau} \geq \Delta_ {\tau , i} - C ^ {k} \Delta_ {\tau , i} = \Delta_ {\tau , i} (1 - C ^ {k}).
$$
Taking the expectation with respect to  $\tau$  gives
$$
\begin{array}{l} \mathbb {E} _ {\tau} [ E r r _ {t + k} ^ {i, \tau} ] = \sum_ {\tau} p _ {\tau} E r r _ {t + k} ^ {i, \tau}, \\ \geq \sum_ {\tau} p _ {\tau} \Delta_ {\tau , i} (1 - C ^ {k}), \\ \geq (1 - C ^ {k}) \mathbb {E} _ {\tau} [ \Delta_ {\tau , i} ]. \\ \end{array}
$$
Now we will find a timestep  $k_{0}$  such that  $\forall k > k_{0}$ ,  $E_{\tau}[Err_{t+k}^{i,\tau}] > D_{i}$ . Let  $k_{i}$  denote the timestep for which  $(1 - C^{k_{i}})E_{\tau}[\Delta_{\tau,i}] = D_{i}$ . Then,
$$
\left(1 - C ^ {k _ {i}}\right) E _ {\tau} \left[ \Delta_ {\tau , i} \right] = D _ {i},
$$
$$
\left(1 - C ^ {k _ {i}}\right) = \frac {D _ {i}}{E _ {\tau} \left[ \Delta_ {\tau , i} \right]},
$$
$$
C ^ {k _ {i}} = \frac {E _ {\tau} [ \Delta_ {\tau , i} ] - D _ {i}}{E _ {\tau} [ \Delta_ {\tau , i} ]},
$$
$$
\boxed {k _ {i} = \frac {1}{\log C} \log \left(\frac {E _ {\tau} [ \Delta_ {\tau , i} ] - D _ {i}}{E _ {\tau} [ \Delta_ {\tau , i} ]}\right).}
$$
So, after time  $t + k_{i}$ ,  $V^{(P)}$  is a better estimate of the value function of the past task  $i$  than the current TD estimates. Then, by letting  $k_{0} = \max_{i} k_{i}$  we have  $\mathbb{E}_{\tau}[Err_{t + k_{0}}^{i,\tau}] > D_{i}, \forall i$ .
Note that the above result depends on the fact that  $k_{i}$  exists, which is only possible when  $E_{\tau}[\Delta_{\tau,i}] - D_{i} > 0$ . We will now show that there exists at least one solution, the fixed point of the permanent value function  $(\mathbb{E}_{\tau}[v_{\tau}])$ , which satisfies this constraint.
$$
\begin{array}{l} E _ {\tau} \left[ \Delta_ {\tau , i} \right] > \left\| \mathbb {E} _ {\tau} \left[ v _ {i} - v _ {\tau} \right] \right\| _ {2} ^ {2}, (\text {U s i n g J e s s e n ' s i n e q u a l i t y}) \\ = \left\| v _ {i} - \mathbb {E} _ {\tau} [ v _ {\tau} ] \right\| _ {2} ^ {2}. \\ \end{array}
$$
$$
\boxed {E _ {\tau} [ \Delta_ {\tau , i} ] - D _ {i} > \| v _ {i} - \mathbb {E} _ {\tau} [ v _ {\tau} ] \| _ {2} ^ {2} - D _ {i} = 0.}
$$
The final step follows from the fact that  $D_{i} = \left\| V^{(P)} - v_{i}\right\|_{2}^{2} = \left\| \mathbb{E}_{\tau}[v_{\tau}] - v_{i}\right\|_{2}^{2}$  for  $V^{(P)}$  fixed point.
Next, we show that our method has tighter upper bound on errors compared with the TD learning algorithm in expectation.
Theorem B.3.2. Under assumption 1, there exists  $k_{0}$  such that  $\forall k\leq k_0,\mathbb{E}_{\tau}\left[\left\| v_{\tau} - V_{t + k}^{(PT)}\right\| _2^2\right]$  has a tighter upper bound compared with  $\mathbb{E}_{\tau}\left[\left\| v_{\tau} - V_{t + k}^{(TD)}\right\| _2^2\right],\forall i.$
Proof. We assume that  $V^{(P)}$  is converged to its fixed point,  $\mathbb{E}_{\tau}[v_{\tau}]$  (Theorem 5). Using sample complexity bounds from Bhandari et al. [6] for TD learning algorithm, we get:
$$
\begin{array}{l} \mathbb {E} _ {\tau} \left[ \left\| v _ {\tau} - V _ {t + k} ^ {(T D)} \right\| _ {2} ^ {2} \right] \leq \mathbb {E} _ {\tau} \left[ \exp \left\{- \left(\frac {(1 - \gamma) ^ {2} \omega}{4}\right) T \right\} \left\| v _ {\tau} - V _ {t} ^ {(T D)} \right\| _ {2} ^ {2} \right], \\ = \exp \left\{- \left(\frac {(1 - \gamma) ^ {2} \omega}{4}\right) T \right\} \mathbb {E} _ {\tau} \left[ \left\| v _ {\tau} - V _ {t} ^ {(T D)} \right\| _ {2} ^ {2} \right]. \\ \end{array}
$$
We know that our algorithm can be interpreted as TD learning with  $V^{(P)}$  as the starting point (see Theorem 1). Using this relationship, we can apply the sample complexity bound to get:
$$
\begin{array}{l} \mathbb {E} _ {\tau} \left[ \left\| v _ {\tau} - V _ {t + k} ^ {(P T)} \right\| _ {2} ^ {2} \right] \leq \mathbb {E} _ {\tau} \left[ \exp \left\{- \left(\frac {(1 - \gamma) ^ {2} \omega}{4}\right) T \right\} \left\| v _ {\tau} - V _ {t} ^ {(P)} \right\| _ {2} ^ {2} \right], \\ = \exp \left\{- \left(\frac {(1 - \gamma) ^ {2} \omega}{4}\right) T \right\} \mathbb {E} _ {\tau} \left[ \left\| v _ {\tau} - V _ {t} ^ {(P)} \right\| _ {2} ^ {2} \right]. \\ \end{array}
$$
For  $k = 0$ , we know that our algorithm has lower error in expectation (see Theorem 6), that is  $\mathbb{E}_{\tau}\left[\left\| v_{\tau} - V_t^{(P)}\right\|_2^2\right] \leq \mathbb{E}_{\tau}\left[\left\| v_{\tau} - V_t^{(TD)}\right\|_2^2\right]$ . Therefore, the error bound for our algorithm is tighter compared with that of TD learning algorithm. As  $k \to \infty$ , the right hand side of both the equations goes to 0, which confirms that both algorithms converge to the true value function.
The theorem shows that the estimates for our algorithm lie within a smaller region compared to that of TD learning estimates. This doesn't guarantee low errors for our algorithms, but we can expect it to be closer to the true value function.
The next theorems contain analytical MSE equations for TD-learning and for our algorithm. Our analysis is similar to that of [42], but we don't assume that rewards are only observed in the final transition. Analytical MSE equations allow calculating the exact MSE that an algorithm would incur at each timestep when applied to a problem, instead of averaging the performance over several seeds, which is the first step towards other analyses such as bias-variance decomposition of value estimates, error bounds and bias-variance bounds, step-size adaptation for the update rule, etc. We use these equations to compare our method against TD-learning when varying the task distribution diameter (Sec. C.2).
We assume that the agent gets  $N$  i.i.d samples  $\langle s, r, s' \rangle$  from task  $\tau$  in each round, where  $s$  is sampled according to the stationary distribution  $d_{\tau}$ . Also, the agent accumulates errors from all the samples and updates its estimates at the end of the round  $t$ . Let  $\mathbb{E}_{\text{seeds}}$  denote the expectation with respect to random seeds, i.e., expectation with respect to the data generation process which generates the data that the agent observes over rounds and within a round.
Theorem B.3.3. Let:
$$
m _ {t} ^ {(T D)} (s) = \mathbb {E} _ {s e e d s} \bigl [ V _ {t} ^ {(T D)} (s) \bigr ],
$$
$$
C o v (V _ {t} ^ {(T D)} (s), V _ {t} ^ {(T D)} (x)) = \Sigma_ {t} ^ {(T D)} (s, x) - m _ {t} ^ {(T D)} (s) m _ {t} ^ {(T D)} (x),
$$
where  $\Sigma_t^{(TD)}(s,x) = \mathbb{E}_{\text{seeds}}[V_t^{(TD)}(s)V_t^{(TD)}(s)]$ . We have:
$$
m _ {t + 1} ^ {(T D)} (s) = m _ {t} ^ {(T D)} (s) + \alpha N d _ {\pi} (s) \Delta_ {t} ^ {(T D)} (s),
$$
$$
\Sigma_ {t + 1} ^ {(T D)} (s, x) = \Sigma_ {t} ^ {(T D)} (s, x) + \alpha \Omega_ {t} ^ {(T D)} (s, x) + \alpha^ {2} \Psi_ {t} ^ {(T D)} (s, x),
$$
and, the mean squared error  $\xi_t^{(TD)}$  at timestep  $t$  is
$$
\begin{array}{l} \xi_ {t + 1} ^ {(T D)} = \xi_ {t} ^ {(T D)} + \alpha \sum_ {s \in \mathcal {S}} d _ {\pi} (s) (- 2 v _ {\pi} (s) \Delta_ {t} ^ {(T D)} (s) + \Omega_ {t} ^ {(T D)} (s, s)) \\ + \alpha^ {2} \sum_ {s \in \mathcal {S}} d _ {\pi} (s) \Psi_ {t} ^ {(T D)} (s, s). \\ \end{array}
$$
$$
\begin{array}{l} m _ {t + 1} ^ {(T D)} (s) = \mathbb {E} _ {s e e d s} \big [ V _ {t + 1} ^ {(T D)} (s) \big ], \\ = \mathbb {E} _ {\text {s e e d s}} \left[ V _ {t} ^ {(T D)} (s) + \alpha \sum_ {k = 1} ^ {N} \mathbb {I} _ {S _ {k} = s} \left(R _ {k} + \gamma V _ {t} ^ {(T D)} \left(S _ {k} ^ {\prime}\right) - V _ {t} ^ {(T D)} (s)\right) \right], \\ = m _ {t} ^ {(T D)} (s) + \alpha \sum_ {k = 1} ^ {N} \mathbb {E} _ {s e e d s} [ \mathbb {I} _ {S _ {k} = s} (R _ {k} + \gamma V _ {t} ^ {(T D)} (S _ {k} ^ {\prime}) - V _ {t} ^ {(T D)} (s)) ], \\ \end{array}
$$
$$
\begin{array}{l} = m _ {t} ^ {(T D)} (s) + \alpha \sum_ {k = 1} ^ {N} \mathbb {E} _ {\text {s e e d s}} \left[ \mathbb {E} [ \mathbb {I} _ {S _ {k} = s} (R _ {k} + \gamma V _ {t} ^ {(T D)} (S _ {k} ^ {\prime}) - V _ {t} ^ {(T D)} (s)) | \text {s e e d} ] \right], \\ = m _ {t} ^ {(T D)} (s) + \alpha \sum_ {k = 1} ^ {N} \mathbb {E} _ {s e e d s} \left[ \widehat {d _ {\pi}} (s) \mathbb {E} [ (R _ {k} + \gamma V _ {t} ^ {(T D)} (S _ {k} ^ {\prime}) - V _ {t} ^ {(T D)} (s)) | s e e d, S _ {k} = s ] \right], \\ = m _ {t} ^ {(T D)} (s) + \alpha \sum_ {k = 1} ^ {N} \mathbb {E} _ {s e e d s} \left[ \underbrace {\widehat {d _ {\pi}} (s) (\widehat {r _ {\pi}} (s) + \gamma (\widehat {\mathcal {P} _ {\pi}} = V _ {t} ^ {(T D)}) (s) - V _ {t} ^ {(T D)} (s))} _ {\text {I n d e p e n d e n t q u a n t i t i e s}} \right], \\ = m _ {t} ^ {(T D)} (s) + \alpha \sum_ {k = 1} ^ {N} d _ {\pi} (s) \left(r _ {\pi} (s) + \gamma \left(\mathcal {P} _ {\pi} m _ {t} ^ {(T D)}\right) (s) - m _ {t} ^ {(T D)} (s)\right) ], \\ = m _ {t} ^ {(T D)} (s) + \alpha N d _ {\pi} (s) \left(r _ {\pi} (s) + \gamma \left(\mathcal {P} _ {\pi} m _ {t} ^ {(T D)}\right) (s) - m _ {t} ^ {(T D)} (s)\right), \\ \end{array}
$$
$$
\boxed {m _ {t + 1} ^ {(T D)} (s) = m _ {t} ^ {(T D)} (s) + \alpha \Delta_ {t} ^ {(T D)} (s),}
$$
where  $\Delta_t^{(TD)}(s) = r_\pi(s) + \gamma(\mathcal{P}_\pi m_t^{(TD)})(s) - m_t^{(TD)}(s)$ . We used the following results in the proof:
1.  $\mathbb{E}_{\text{seeds}}[\widehat{d_{\pi}}(s)\widehat{r_{\pi}}(s)] = \mathbb{E}_{\text{seeds}}[\widehat{d_{\pi}}(s)]\mathbb{E}_{\text{seeds}}[\widehat{r_{\pi}}(s)] = d_{\pi}(s)r_{\pi}(s).$
2.  $\mathbb{E}_{\text{seeds}}[\widehat{d_{\pi}}(s)\gamma(\widehat{\mathcal{P}_{\pi}}V_t^{(TD)})(s)] = \gamma\mathbb{E}_{\text{seeds}}[\widehat{d_{\pi}}(s)]\mathbb{E}_{\text{seeds}}[(\widehat{\mathcal{P}_{\pi}}V_t^{(TD)})(s)]$
$$
\begin{array}{l} = \gamma d _ {\pi} (s) \sum_ {s ^ {\prime}} \mathbb {E} _ {s e e d s} [ \widehat {\mathcal {P} _ {\pi}} (s ^ {\prime} | s) V _ {t} ^ {(T D)} (s ^ {\prime}) ], \\ = \gamma d _ {\pi} (s) \sum_ {s ^ {\prime}} \mathcal {P} _ {\pi} (s ^ {\prime} | s) m _ {t} ^ {(T D)} (s ^ {\prime}) ], \\ = \gamma d _ {\pi} (s) \left(\mathcal {P} _ {\pi} m _ {t} ^ {(T D)}\right) (s). \\ \end{array}
$$
3.  $\mathbb{E}_{\text{seeds}}[\widehat{d_{\pi}}(s)V_t^{(TD)}(s)] = \mathbb{E}_{\text{seeds}}[\widehat{d_{\pi}}(s)]\mathbb{E}_{\text{seeds}}[V_t^{(TD)}(s)] = d_{\pi}(s)m_t^{(TD)}(s)$ .
Now consider  $\Sigma_{t + 1}^{(TD)}(s,x)$
$$
\begin{array}{l} \Sigma_ {t + 1} ^ {(T D)} (s, x) = \mathbb {E} _ {\text {s e e d s}} \left[ V _ {t + 1} ^ {(T D)} (s) V _ {t + 1} ^ {(T D)} (x) \right], \\ = \mathbb {E} _ {\text {s e e d s}} \left[ \left(V _ {t} ^ {(T D)} (s) + \alpha \sum_ {k = 1} ^ {N} \mathbb {I} _ {S _ {k} = s} \left(R _ {k} + \gamma V _ {t} ^ {(T D)} \left(S _ {k} ^ {\prime}\right) - V _ {t} ^ {(T D)} (s)\right) V _ {t + 1} ^ {(T D)} (x)\right) \right. \\ \left. \left(V _ {t} ^ {(T D)} (x) + \alpha \sum_ {l = 1} ^ {N} \mathbb {I} _ {S _ {l} = x} \left(R _ {l} + \gamma V _ {t} ^ {(T D)} \left(X _ {l} ^ {\prime}\right) - V _ {t} ^ {(T D)} (x)\right)\right) \right], \\ = \mathbb {E} _ {\text {s e e d s}} \left[ V _ {t} ^ {(T D)} (s) V _ {t} ^ {(T D)} (x) \right] \\ + \alpha \mathbb {E} _ {\text {s e e d s}} \left[ V _ {t} ^ {(T D)} (s) \left(\sum_ {l = 1} ^ {N} \mathbb {I} _ {S _ {l} = x} \left(R _ {l} + \gamma V _ {t} ^ {(T D)} \left(X _ {l} ^ {\prime}\right) - V _ {t} ^ {(T D)} (x)\right)\right) \right] (9) \\ + \alpha \mathbb {E} _ {\text {s e e d s}} \left[ V _ {t} ^ {(T D)} (x) \left(\sum_ {k = 1} ^ {N} \mathbb {I} _ {S _ {k} = s} \left(R _ {k} + \gamma V _ {t} ^ {(T D)} \left(S _ {k} ^ {\prime}\right) - V _ {t} ^ {(T D)} (s)\right)\right) \right] (10) \\ + \alpha^ {2} \mathbb {E} _ {\text {s e e d s}} \left[ \left(\sum_ {k = 1} ^ {N} \mathbb {I} _ {S _ {k} = s} \left(R _ {k} + \gamma V _ {t} ^ {(T D)} \left(S _ {k} ^ {\prime}\right) - V _ {t} ^ {(T D)} (s)\right)\right) \right. \\ \left. \left(\sum_ {l = 1} ^ {N} \mathbb {I} _ {S _ {l} = x} \left(R _ {l} + \gamma V _ {t} ^ {(T D)} \left(X _ {l} ^ {\prime}\right) - V _ {t} ^ {(T D)} (x)\right)\right) \right] (11) \\ \end{array}
$$
Next, we will tackle each part separately. We will begin with Eq (9):