SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Oscar Li $^{\S 1}$ , James Harrison $^{\diamond}$ , Jascha Sohl-Dickstein $^{\diamond}$ , Virginia Smith $^{\S}$ , Luke Metz $^{\diamond 2}$
Machine Learning Department, School of Computer Science Carnegie Mellon University
Unrolled computation graphs are prevalent throughout machine learning but present challenges to automatic differentiation (AD) gradient estimation methods when their loss functions exhibit extreme local sensitivity, discontinuity, or blackbox characteristics. In such scenarios, online evolution strategies methods are a more capable alternative, while being more parallelizable than vanilla evolution strategies (ES) by interleaving partial unrolls and gradient updates. In this work, we propose a general class of unbiased online evolution strategies methods. We analytically and empirically characterize the variance of this class of gradient estimators and identify the one with the least variance, which we term Noise-Reuse Evolution Strategies (NRES). Experimentally<sup>3</sup>, we show NRES results in faster convergence than existing AD and ES methods in terms of wall-clock time and number of unroll steps across a variety of applications, including learning dynamical systems, meta-training learned optimizers, and reinforcement learning.
First-order optimization methods are a foundational tool in machine learning. With many such methods (e.g., SGD, Adam) available in existing software, ML training often amounts to specifying a computation graph of learnable parameters and computing some notion of gradients to pass into an off-the-shelf optimizer. Here, unrolled computation graphs (UCGs), where the same learnable parameters are repeatedly applied to transition a dynamical system's inner state, have found their use in various applications such as recurrent neural networks [1, 2], meta-training learned optimizers [3, 4], hyperpameter tuning [5, 6], dataset distillation [7, 8], and reinforcement learning [9, 10].
While a large number of automatic differentiation (AD) methods exist to estimate gradients in UCGs [11], they often perform poorly over loss landscapes with extreme local sensitivity and cannot handle black-box computation dynamics or discontinuous losses [12, 3, 13]. To handle these shortcomings, evolution strategies (ES) have become a popular alternative to produce gradient estimates in UCGs [14]. ES methods convolve the (potentially pathological or discontinuous) loss surface with a Gaussian distribution in the learnable parameter space, making it smoother and infinitely differentiable. Unfortunately, vanilla ES methods cannot be applied online<sup>4</sup> — the computation must reach the end of the graph to produce a gradient update, thus incurring large update latency for long UCGs. To address this, a recently proposed approach, Persistent Evolution Strategies [15] (PES), samples a new Gaussian noise in every truncation unroll and accumulates the past sampled noises to get rid of the estimation bias in its online application.
In this work, we investigate the coupling of the noise sampling frequency and the gradient estimation frequency in PES. By decoupling these two values, we arrive at a more general class of unbiased,
$^{1}$ Correspondence to: <oscarli@cmu.edu>.  $^{2}$ Now at OpenAI.
$^{3}$ Code available at https://github.com/OscarcarLi/Noise-Reuse-Evolution-Strategies.
4 Online here means a method can produce gradient estimates using only a truncation window of an unrolled computation graph instead of the full graph, thus allowing the interleaving of partial unrolls and gradient updates.
Figure 1: (a) The pathological loss surface in the learned optimizer task (Sec. 5.2) along a random  $\epsilon$  direction; such surfaces are common in UCGs but can make automatic differentiation methods unusable, leading to the recent development of evolution strategies methods. (b) Comparison of properties of different evolution strategies methods. Unlike prior online ES methods, NRES produces both unbiased and low-variance gradient estimates.
online ES gradient estimators. Through a variance characterization of these estimators, we find that the one which provably has the lowest variance in fact reuses the same noise for the entire time horizon (instead of over a single truncation window as in PES). We name this method Noise-Reuse Evolution Strategies (NRES). In addition to being simple to implement, NRES converges faster than PES across a wide variety of applications due to its reduced variance. Overall, we make the following contributions:
- We propose a class of unbiased online evolution strategies gradient estimators for unrolled computation graphs that generalize Persistent Evolution Strategies [15].
- We analytically and empirically characterize the variance of this class of estimators and identify the lowest-variance estimator which we name Noise-Reuse Evolution Strategies (NRES).
- We identify the connection between NRES and the existing offline ES method FullES and show that NRES is a better alternative to FullES both in terms of parallelizability and variance.
- We demonstrate that NRES can provide optimization convergence speedups (up to  $5 - 60\times$ ) over AD/ES baselines in terms of wall-clock time and number of unroll steps in applications of 1) learning dynamical systems, 2) meta-training learned optimizers, and 3) reinforcement learning.
2 Online Evolution Strategies: Background and Related Work
Problem setup. Unrolled computation graphs (UCGs) [15] are common in applications such as training recurrent neural networks, meta-training learned optimizers, and learning reinforcement learning policies, where the same set of parameters are repeatedly used to update the inner state of some system. We consider general UCGs where the inner state  $s_t \in \mathbb{R}^p$  is updated with learnable parameters  $\theta \in \mathbb{R}^d$  through transition functions:  $\{f_t : \mathbb{R}^p \times \mathbb{R}^d \to \mathbb{R}^p\}_{t=1}^T$ ,  $s_t = f_t(s_{t-1}; \theta)$  for  $T$  time steps starting from an initial state  $s_0$ . At each time step  $t \in \{1, \dots, T\}$ , the state  $s_t$  incurs a loss  $L_t^s(s_t)$ . As the loss  $L_t^s$  depends on  $t$  applications of  $\theta$ , we make this dependence more explicit with a loss function  $L_t : \mathbb{R}^{dt} \to \mathbb{R}$  and  $L_t([\theta]_{\times t}) := L_t^s(s_t)^5$ . We aim to minimize the average loss over all  $T$  time steps unrolled under the same  $\theta$ ,  $\min_\theta L([\theta]_{\times T})$ , where
$$
L \left(\theta_ {1}, \dots , \theta_ {T}\right) := \frac {1}{T} \sum_ {t = 1} ^ {T} L _ {t} \left(\theta_ {1}, \dots , \theta_ {t}\right). \tag {1}
$$
Loss properties. Despite the existence of many automatic differentiation (AD) techniques to estimate gradients in UCGs [11], there are common scenarios where they are undesirable: 1) Loss surfaces with extreme local sensitivity: With large number of unrolls in UCGs, the induced loss surface is prone to high degrees of sharpness and many suboptimal local minima (see Figure 1). This issue is particularly prevalent when the underlying dynamical system is chaotic under the parameter  $(\theta)$  of interest [13], e.g. in model-based control [12] and meta-learning [3]. In such cases, naively following the gradient may either  $a$  fail to converge under the normal range of learning rates (because of the conflicting gradient directions) or  $b$  converge to highly suboptimal solutions using a tuned, yet much smaller learning rate. 2) Black-box or discontinuous losses: As AD methods require defining a Jacobian-vector product (forward-mode) or a vector-Jacobian product (reverse-mode) for every elementary operation in the computation graph, they cannot be applied when the UCG's inner dynamics are inaccessible (e.g., model-free reinforcement learning) or the loss objectives (e.g. accuracy) are piecewise constant (zero gradients).
Evolution Strategies. Due to the issues with AD methods described above, a common alternative is to use evolution strategies (ES) to estimate gradients. Here, the original loss function is convolved with an isotropic Gaussian distribution in the space of  $\theta$ , resulting in an infinitely differentiable loss
$^{5}[\theta]_{\times a}$ ,  $a \in \mathbb{Z}_{\geq 0}$  denotes  $a$  copies of  $\theta$ , for example:  $L_{t}([\theta]_{\times a}, [\theta']_{\times t - a}) := L_{t}(\underbrace{\theta, \ldots, \theta}_{a \text{ times}}, \underbrace{\theta', \ldots, \theta'}_{(t - a) \text{ times}})$ .
(a) Online Evolution Strategies Protocol
(b) How different unbiased OES methods share noise?
Figure 2: (a) Illustration of step-unlocked online ES workers working independently at different truncation windows. Here a central server sends  $\theta$  (whose gradient to be estimated) to each worker and receives the estimates over partial unrolls from each. The averaged gradient can then be used in a first-order optimization algorithm. (b) Comparison of the noise sharing mechanisms of PES,  $\mathrm{GPES}_K$ , and NRES (ours). Unlike PES (and  $\mathrm{GPES}_{K\neq T}$ ) which samples a new noise in every (some) truncation window and needs to accumulate the noise, NRES only samples noise once at the beginning of an episode and reuses the noise for the full episode.
function with lower sharpness and fewer local minima than before ( $\sigma > 0$  is a hyperparameter):
$$
\theta \mapsto \mathbb {E} _ {\epsilon \sim \mathcal {N} (0, \sigma^ {2} I _ {d \times d})} L ([ \theta + \epsilon ] _ {\times T}). \tag {2}
$$
An unbiased gradient estimator of (2) is given by the likelihood ratio gradient estimator [16]:  $\frac{1}{\sigma^2} L([\theta + \epsilon]_{\times T})\epsilon$ . This estimator only requires the loss evaluation (hence is zeroth-order) but not an explicit computation of the gradient, thus being applicable in cases when the gradients are noninformative (chaotic or piecewise constant loss) or not directly computable (black-box loss). To reduce the variance, antithetic sampling is used and we call this estimator FullES (Algorithm 3 in the Appendix):
$$
\operatorname {F u l l E S} (\theta) := \frac {1}{2 \sigma^ {2}} \left[ \frac {1}{T} \sum_ {i = 1} ^ {T} \left(L _ {i} \left(\left[ \theta + \epsilon \right] _ {\times i}\right) - L _ {i} \left(\left[ \theta - \epsilon \right] _ {\times i}\right) \right] \epsilon . \right. \tag {3}
$$
The term Full highlights that this estimator can only produce a gradient estimate after a full sequential unroll from  $t = 0$  to  $T$ . We call such a full unroll an episode following the reinforcement learning terminology. FullES can be parallelized [14] by averaging  $N$  parallel gradient estimates using i.i.d.  $\epsilon$ 's, but is not online and can result in substantial latency between gradient updates when  $T$  is large.
Truncated Evolution Strategies. To make FullES online, Metz et al. [3] take inspiration from truncated backpropagation through time (TBPTT) and propose the algorithm TES (see Algorithm 6 in the Appendix). Unlike the stateless estimator FullES, TES is stateful: TES starts from a saved state  $s$  (from the previous iteration) and draws a new  $\epsilon_{i}$  for antithetic unrolling. To make itself online, TES only unrolls for a truncation window of  $W$  steps for every gradient estimate, thus reducing the latency from  $O(T)$  to  $O(W)$ . Analytically,
$$
\operatorname {T E S} (\theta) := \frac {1}{2 \sigma^ {2} W} \sum_ {i = 1} ^ {W} \left[ L _ {\tau + i} \left(\left[ \theta \right] _ {\times \tau}, \left[ \theta + \boldsymbol {\epsilon} _ {\left(\tau / W\right) + 1} \right] _ {\times i}\right) - L _ {\tau + i} \left(\left[ \theta \right] _ {\times \tau}, \left[ \theta - \boldsymbol {\epsilon} _ {\left(\tau / W\right) + 1} \right] _ {\times i}\right) \right] \boldsymbol {\epsilon} _ {\left(\tau / W\right) + 1}. \tag {4}
$$
Here, besides the Gaussian random variables  $\epsilon_i\big|_{i = 1}^{T / W}\stackrel {\mathrm{iid}}{\sim}\mathcal{N}(0,\sigma^2 I_{d\times d})$ , the time step  $\pmb{\tau}$  which TES starts from is also a random variable drawn from the uniform distribution  $\pmb{\tau}\sim$  Unif  $\{0,W,\dots ,T-$ $W\} ^6$ . It is worth noting that Vicol et al. [15] who also analyze online ES estimators do not take the view that the time step  $\pmb{\tau}$  an online ES estimator starts from is a random variable; as such their analyses do not fully reflect the "online" nature captured in our work. When multiple online ES Workers (e.g., TES workers) run in parallel, different workers will work at different i.i.d. time steps  $\pmb{\tau}_i$  (which we call step-unlocked workers) (see Figure 2(a)). We provide the pseudocode for creating step-unlocked workers and for general online ES learning in Algorithm 4 and 5 in the Appendix.
TES is a biased gradient estimator of (2). Note that in (4), only the  $\theta$ 's in the current length- $W$  truncation window receive antithetic perturbations, thus ignoring the impact of the earlier  $\theta$ 's up to time step  $\tau$ . Due to this bias, optimization using TES typically doesn't converge to optimal solutions.
Persistent Evolution Strategies. To resolve the bias of TES, Vicol et al. [15] recognize that TES samples a new noise in every truncation window and modifies the smoothing objective into:
$$
\theta \mapsto \mathbb {E} _ {\{\epsilon_ {i} \}} L ([ \theta + \epsilon_ {1} ] _ {\times W}, \dots , [ \theta + \epsilon_ {T / W} ] _ {\times W}). \tag {5}
$$
The random variable  $\tau$  will always be sampled from this uniform distribution. In addition, we will assume the time horizon  $T$  can be evenly divided into truncation windows of length  $W$ , i.e.  $T \equiv 0 \pmod{W}$ .
Algorithm 1 Persistent Evolution Strategies [15]
class PESWorker(OnlineESWorker):
def __init__(self, W): self.  $\pmb {\tau} = 0$  self.s  $s^{+} = s_{0}$  self.s  $s^{-} = s_{0}$  self.W=W; self.  $\pmb {\xi} = \pmb {0}\in \mathbb{R}^d$
def gradient Estimate(self,  $\theta$
sample at every truncation window  $\epsilon \sim \mathcal{N}(\mathbf{0},\sigma^2 I_{d\times d})$  # this is  $\epsilon_{(\tau /W) + 1}$  self.  $\pmb {\xi} + = \pmb{\epsilon}$  # now self.  $\pmb {\xi} = \sum_{i = 1}^{\tau /W}\epsilon_i + \epsilon_{\tau /W + 1}$
$(s^{+}, s^{-}) = (\text{self.} s^{+}, \text{self.} s^{-})$ $L_{\text{sum}}^{+} = 0; L_{\text{sum}}^{-} = 0$
for  $i$  in range(1, self.  $W + 1$ ):
$$
s ^ {+} = f _ {\text {s e l f .} \tau + i} \left(s ^ {+}, \theta + \epsilon\right)
$$
$$
s ^ {-} = f _ {\text {s e l f .} \tau + i} \left(s ^ {-}, \theta - \epsilon\right)
$$
$$
L _ {\text {s u m}} ^ {+} + = L _ {\text {s e l f .} \tau + i} ^ {s} (s ^ {+})
$$
$$
L _ {\text {s u m}} ^ {-} + = L _ {\text {s e l f .} \tau + i} ^ {s} (s ^ {-})
$$
$g = (L_{\mathrm{sum}}^{+} - L_{\mathrm{sum}}^{-}) / (2\sigma^{2}\cdot \mathrm{self}.W)$  self.  $\pmb{\xi}$  self.s+=s+; self.s-=s- self.t= self.t+W
if self.  $\pmb{\tau}\geq T$  #reset at the end self.  $\pmb {\tau} = 0$  self.s+  $s_0$  self.s-  $s_0$  self.  $\pmb {\xi} = \mathbf{0}$
return  $g$
Algorithm 2 Noise-Reuse Evolution Strategies (ours)
class NRESWorker(OnlineESWorker):
def init_self, W): self.  $\pmb {\tau} = 0$  self.  $s^+ = s_0$  self.  $s^{-} = s_{0}$  self.  $W = W$
if self.  $\pmb{\tau} = =$  0: # only sample at beginning  $\epsilon \sim \mathcal{N}(\mathbf{0},\sigma^2 I_{d\times d})$  self.e= e #reuse for this episode
$$
s ^ {+} = f _ {\text {s e l f .} \tau + i} \left(s ^ {+}, \theta + \text {s e l f .} \epsilon\right)
$$
$$
s ^ {-} = f _ {\text {s e l f .} \tau + i} \left(s ^ {-}, \theta - \text {s e l f .} \epsilon\right)
$$
$g = (L_{\mathrm{sum}}^{+} - L_{\mathrm{sum}}^{-}) / (2\sigma^{2}\cdot \mathrm{self}.W)$  self.e self.s  $s^+ = s^+$  ; self.s  $= s^{-}$  self.  $\pmb{\tau} =$  self.  $\pmb{\tau} + W$
if self.  $\pmb{\tau}\geq T$  :#reset at the end self.  $\pmb {\tau} = 0$  self.s  $^+$ $s_0$  ;self.s-  $= s_{0}$
They show that an unbiased gradient estimator of 5 is given by (see Algorithm 1):
$$
\begin{array}{l} \mathrm {P E S} (\theta) := \frac {1}{2 \sigma^ {2} W} \sum_ {i = 1} ^ {W} \left[ L _ {\tau + i} ([ \theta + \epsilon_ {1} ] _ {\times W}, \dots , [ \theta + \epsilon_ {(\tau / W) + 1} ] _ {\times i}) \right. \\ \left. - L _ {\tau + i} \left(\left[ \theta - \epsilon_ {1} \right] _ {\times W}, \dots , \left[ \theta - \epsilon_ {\left(\tau / W\right) + 1} \right] _ {\times i}\right) \right] \left(\sum_ {j = 1} ^ {\tau / W} \epsilon_ {j}\right) + \epsilon_ {\tau / W + 1}), \tag {6} \\ \end{array}
$$
with randomness in both  $\{\epsilon_i\}$  and  $\pmb{\tau}$ . To eliminate the bias of TES, instead of multiplying only with the current epsilon  $\epsilon_{\tau /W + 1}$ , PES multiplies it with the cumulative sum of all the different  $iid$  noise sampled so far (self. $\xi$  in PESWorker). As we shall see in the next section, this accumulation of noise terms provably results in higher variance, making PES less desirable in practice. We contrast the noise sampling properties of our proposed methods with PES in Figure 1(b).
Hysteresis. When online gradient estimators are used in training, they often suffer from hysteresis, or history dependence, as a result of the parameters  $\theta$  changing between adjacent unrolls. That is, the parameter value  $\theta_0$  that a worker uses in the current truncation window is not the same parameter  $\theta_{-1}$  that was used in the previous window. This effect is often neglected [15], under an assumption that  $\theta$  is updated slowly. To the best of our knowledge, [17] is the only work to analyze the convergence of an online gradient estimator under hysteresis. In the following theoretical analysis, we assume all online gradient estimates are computed without hysteresis in order to isolate the problem. However, in Section 5, we show empirically that even under the impact of hysteresis, our proposed online estimator NRES can outperform non-online methods (e.g., FullES) which don't suffer from hysteresis.
3 A New Class of Unbiased Online Evolution Strategies Methods
As shown in Section 2, TES and PES both sample a new noise perturbation  $\epsilon$  for every truncation window to produce gradient estimates. Here we note that the frequency of noise-sharing (new noise every truncation window of size  $W$ ) is fixed to the frequency of gradient estimates (a gradient estimate every truncation window of size  $W$ ). However, the former is a choice of the smoothing objective (5), while the latter is often a choice of how much gradient update latency the user can tolerate. In this section we break this coupling by introducing a general class of gradient estimators that encompass PES. We then analyze these estimators' variance to identify the one with the least variance.
Generalized Persistent Evolution Strategies (GPES). For a given fixed truncation window size  $W$ , we consider all noise-sharing periods  $K$  that are multiples of  $W$ ,  $K = cW$  for  $c \in \mathbb{Z}^+$ ,  $c \leq T / W$ .  $K$  being a multiple of  $W$  ensures that within each truncation window, only a single  $\epsilon$  is used. When
$K = W$ , we recover the PES algorithm. However, when  $K$  is larger than  $W$ , the same noise will be used across adjacent truncation windows (Figure 2(b)). With a new noise sampled every  $K$  unroll steps, we define the  $K$ -smoothed loss objective as the function:
$$
\theta \mapsto \mathbb {E} _ {\left\{\epsilon_ {i} \right\}} L \left(\left[ \theta + \epsilon_ {1} \right] _ {\times K}, \dots , \left[ \theta + \epsilon_ {\lceil T / K \rceil} \right] _ {\times r (T, K)}\right), ^ {7} \tag {7}
$$
where  $\mathrm{r}:(\mathbb{Z}^{+})^{2}\to \mathbb{Z}^{+}$  is the modified remainder function such that  $\mathrm{r}(x,y)$  is the unique integer  $n\in [1,y]$  where  $x = qy + n$  for some integer  $q$ . This extra notation allows for the possibility that  $T$  is not divisible by  $K$  and the last noise  $\epsilon_{[T / K]}$  is used for only  $\mathrm{r}(T,K) < K$  steps.
We now give the analytic form of an unbiased gradient estimator of the resulting smoothed loss.
Lemma 1. An unbiased gradient estimator for the  $K$ -smoothed loss is given by
$$
\begin{array}{l} \operatorname {G P E S} _ {K} (\theta) := \frac {1}{2 \sigma^ {2} W} \sum_ {j = 1} ^ {W} \left[ L _ {\tau + j} ([ \theta + \epsilon_ {1} ] _ {\times K}, \dots , [ \theta + \epsilon_ {\lfloor \tau / K \rfloor + 1} ] _ {\times \mathrm {r} (\tau + j, K)}) \right. \\ \left. - L _ {\tau + j} \left(\left[ \theta - \epsilon_ {1} \right] _ {\times K}, \dots , \left[ \theta - \epsilon_ {\lfloor \tau / K \rfloor + 1} \right] _ {\times r (\tau + j, K)}\right) \right] \cdot \left(\sum_ {i = 1} ^ {\lfloor \tau / K \rfloor + 1} \epsilon_ {i}\right), \\ \end{array}
$$
with randomness in  $\pmb{\tau}$  and  $\{\pmb {\epsilon}_i\}_{i = 1}^{[T / K]}$
$\mathbf{GPES}_K$  algorithm. Here, for the truncation window starting at step  $\pmb{\tau}$ , the noise  $\epsilon_{[\tau /K] + 1}$  is used as the antithetic perturbation to unroll the system. If  $\pmb{\tau}$  is not divisible by  $K$ , then this noise has already been sampled at time step  $t = \lfloor \tau /K\rfloor \cdot K$  in an earlier truncation window. Therefore, to know what noise to apply at this truncation window, we need to remember the last used  $\pmb{\epsilon}$  and update it when  $\pmb{\tau}$  becomes divisible by  $K$ . We provide the algorithm for the  $\mathrm{GPES}_K$  gradient estimator in Algorithm 7 in the Appendix. Note that  $\mathrm{GPES}_{K = W}$  is the same as the PES algorithm.
Variance Characterization of  $\mathbf{GPES}_K$ . With this generalized class of gradient estimators  $\mathrm{GPES}_K$ , one might wonder how to choose the value of  $K$ . Since each estimator is an unbiased gradient estiamtor with respect to its smoothed objective, we compare the variance of these estimators as a function of  $K$ . To do this analytically, we make some simplifying assumptions:
Assumption 2. For a given  $\theta \in \mathbb{R}^d$  and  $t \in [T] := \{1, \dots, T\}$ , there exists a set of vectors  $\{g_i^t \in \mathbb{R}^d\}_{i=1}^t$ , such that for any  $\{v_i \in \mathbb{R}^d\}_{i=1}^t$ , the following equality holds:
$$
L _ {t} \left(\theta + v _ {1}, \theta + v _ {2}, \dots , \theta + v _ {t}\right) - L _ {t} \left(\theta - v _ {1}, \theta - v _ {2}, \dots , \theta - v _ {t}\right) = 2 \sum_ {i = 1} ^ {t} \left(v _ {i}\right) ^ {\top} \left(g _ {i} ^ {t}\right) \tag {8}
$$
Remark 3. This assumption is more general than the quadratic  $L_{t}$  assumption made in [15] (explanation see Appendix D). Here one can roughly understand  $g_{i}^{t}$  as time step  $t$ 's smoothed loss's partial derivative with respect to the  $i$ -th application of  $\theta$ . For notational convenience, we let  $g^{t} \coloneqq \sum_{i=1}^{t} g_{i}^{t}$  (roughly the total derivative of smoothed step-  $t$  loss with respect to  $\theta$ ) and  $g_{K,j}^{t} \coloneqq \sum_{i = K\cdot (j - 1) + 1}^{\min\{t,K\cdot j\}} g_{i}^{t}$  for  $j \in \{1,\dots,\lceil t / K\rceil\}$  (roughly the sum of partial derivatives of smoothed step-  $t$  loss with respect to all  $\theta$ 's in the  $j$ -th noise-sharing window of size  $K$  (the last window might be shorter)).
With this assumption in place, we first consider the case when  $W = 1$  and  $K = cW = c$  for  $c \in [T]$ . In this case, the  $\mathrm{GPES}_K$  estimator can be simplified into the following form:
Lemma 4. Under Assumption 2, when  $W = 1$ ,  $\mathrm{GPES}_{K = c}(\theta) = \frac{1}{\sigma^2}\sum_{j = 1}^{\lfloor \tau /c\rfloor +1}\left(\sum_{i = 1}^{\lfloor \tau /c\rfloor +1}\epsilon_i\right)\epsilon_j^\top g_{c,j}^{\tau +1}$ .
With this simplified form, we can now characterize the variance of the estimator  $\mathrm{GPES}_{K = c}(\theta)$ . Since it's a random vector, we analytically derive its total variance (trace of covariance matrix)  $\operatorname {tr}(\operatorname {Cov}[\operatorname {GPES}_K = c(\theta)])$ .
Theorem 5. When  $W = 1$  and under Assumption 2, for integer  $c \in [T]$ ,