Title: Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies
Venue: NeurIPS 2023
Accepted: yes
Keywords: Evolution Strategies

unrolled computation graph

online gradient estimation

variance reduction

stochastic gradient estimation
Abstract: Unrolled computation graphs are prevalent throughout machine learning but present challenges to automatic differentiation (AD) gradient estimation methods when their loss functions exhibit extreme local sensitivtiy, discontinuity, or blackbox characteristics. In such scenarios, online evolution strategies methods are a more capable alternative, while being more parallelizable than vanilla evolution strategies (ES) by interleaving partial unrolls and gradient updates. In this work, we propose a general class of unbiased online evolution strategies methods. We analytically and empirically characterize the variance of this class of gradient estimators and identify the one with the least variance, which we term Noise-Reuse Evolution Strategies (NRES). Experimentally, we show NRES results in faster convergence than existing AD and ES methods in terms of wall-clock time and number of unroll steps across a variety of applications, including learning dynamical systems, meta-training learned optimizers, and reinforcement learning.

Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies

Abstract

Unrolled computation graphs are prevalent throughout machine learning but present challenges to automatic differentiation (AD) gradient estimation methods when their loss functions exhibit extreme local sensitivtiy, discontinuity, or blackbox characteristics. In such scenarios, online evolution strategies methods are a more capable alternative, while being more parallelizable than vanilla evolution strategies (ES) by interleaving partial unrolls and gradient updates. In this work, we propose a general class of unbiased online evolution strategies methods. We analytically and empirically characterize the variance of this class of gradient estimators and identify the one with the least variance, which we term Noise-Reuse Evolution Strategies (NRES). Experimentally 3 , we show NRES results in faster convergence than existing AD and ES methods in terms of wall-clock time and number of unroll steps across a variety of applications, including learning dynamical systems, meta-training learned optimizers, and reinforcement learning.

1 Introduction

First-order optimization methods are a foundational tool in machine learning. With many such methods (e.g., SGD, Adam) available in existing software, ML training often amounts to specifying a computation graph of learnable parameters and computing some notion of gradients to pass into an off-the-shelf optimizer. Here, unrolled computation graphs (UCGs), where the same learnable parameters are repeatedly applied to transition a dynamical system's inner state, have found their use in various applications such as recurrent neural networks [1, 2], meta-training learned optimizers [3, 4], hyperpameter tuning [5, 6], dataset distillation [7, 8], and reinforcement learning [9, 10].

While a large number of automatic differentiation (AD) methods exist to estimate gradients in UCGs [11], they often perform poorly over loss landscapes with extreme local sensitivity and cannot handle black-box computation dynamics or discontinuous losses [12, 3, 13]. To handle these shortcomings, evolution strategies (ES) have become a popular alternative to produce gradient estimates in UCGs [14]. ES methods convolve the (potentially pathological or discontinuous) loss surface with a Gaussian distribution in the learnable parameter space, making it smoother and infinitely differentiable. Unfortunately, vanilla ES methods cannot be applied online 4 -the computation must reach the end of the graph to produce a gradient update, thus incurring large update latency for long UCGs. To address this, a recently proposed approach, Persistent Evolution Strategies [15] ( PES ), samples a new Gaussian noise in every truncation unroll and accumulates the past sampled noises to get rid of the estimation bias in its online application.

In this work, we investigate the coupling of the noise sampling frequency and the gradient estimation frequency in PES . By decoupling these two values, we arrive at a more general class of unbiased,

1.0

0.5

0.0

0.5

1.0

0.622

0.623

0.624

0.625

0.626

L

([ +

]×

T

)

Learned Optimizer

T

=1000

(a)

(b)

method

online?

unbiased?

FullES

✗

✓

✓

TES

[3]

✓

✗

✓

PES

[15]

✓

✓

✗

NRES

(ours)

✓

✓

✓

online ES gradient estimators. Through a variance characterization of these estimators, we find that the one which provably has the lowest variance in fact reuses the same noise for the entire time horizon (instead of over a single truncation window as in PES ). We name this method Noise-Reuse Evolution Strategies ( NRES ). In addition to being simple to implement, NRES converges faster than PES across a wide variety of applications due to its reduced variance. Overall, we make the following contributions:

We propose a class of unbiased online evolution strategies gradient estimators for unrolled computation graphs that generalize Persistent Evolution Strategies [15].

We analytically and empirically characterize the variance of this class of estimators and identify the lowest-variance estimator which we name Noise-Reuse Evolution Strategies ( NRES ).

We identify the connection between NRES and the existing offline ES method FullES and show that NRES is a better alternative to FullES both in terms of parallelizability and variance.

We demonstrate that NRES can provide optimization convergence speedups (up to 5-60 × ) over AD/ES baselines in terms of wall-clock time and number of unroll steps in applications of 1) learning dynamical systems, 2) meta-training learned optimizers, and 3) reinforcement learning.

2 Online Evolution Strategies: Background and Related Work

Problem setup. Unrolled computation graphs (UCGs) [15] are common in applications such as training recurrent neural networks, meta-training learned optimizers, and learning reinforcement learning policies, where the same set of parameters are repeatedly used to update the inner state of some system. We consider general UCGs where the inner state s t ∈ R p is updated with learnable parameters θ ∈ R d through transition functions: { f t : R p × R d → R p } T t =1 , s t = f t ( s t -1 ; θ ) for T time steps starting from an initial state s 0 . At each time step t ∈ { 1 , . . . , T } , the state s t incurs a loss L s t ( s t ) . As the loss L s t depends on t applications of θ , we make this dependence more explicit with a loss function L t : R dt → R and L t ([ θ ] × t ) := L s t ( s t ) 5 . We aim to minimize the average loss over all T time steps unrolled under the same θ , min θ L ([ θ ] × T ) , where

Loss properties. Despite the existence of many automatic differentiation (AD) techniques to estimate gradients in UCGs [11], there are common scenarios where they are undesirable: 1) Loss surfaces with extreme local sensitiviy : With large number of unrolls in UCGs, the induced loss surface is prone to high degrees of sharpness and many suboptimal local minima (see Figure 1). This issue is particularly prevalent when the underlying dynamical system is chaotic under the parameter ( θ ) of interest [13], e.g. in model-based control [12] and meta-learning [3]. In such cases, naively following the gradient may either a) fail to converge under the normal range of learning rates (because of the conflicting gradient directions) or b) converge to highly suboptimal solutions using a tuned, yet much smaller learning rate. 2) Black-box or discontinuous losses : As AD methods require defining a Jacobian-vector product (forward-mode) or a vector-Jacobian product (reverse-mode) for every elementary operation in the computation graph, they cannot be applied when the UCG's inner dynamics are inaccessible (e.g., model-free reinforcement learning) or the loss objectives (e.g. accuracy) are piecewise constant (zero gradients).

Evolution Strategies. Due to the issues with AD methods described above, a common alternative is to use evolution strategies (ES) to estimate gradients. Here, the original loss function is convolved with an isotropic Gaussian distribution in the space of θ , resulting in an infinitely differentiable loss

θ

Worker

(1)

1

N

N

∑

i

=1

g

(

i

)

/uni23DF

→

s

τ

N

+1→

↑

θ

↑

…→

s

τ

N

+

W

↑

↑

LτN

+1

LτN

+

W

s

τ

N

g

(

N

)

g

(1)

Horizon length

T

Truncation window

length

W

/uni03F5

Online Evolution Strategies Protocol

Worker

(

N

)

+

θ

/uni03F5

+

(a)

Sample

Sample

PESWorker

/uni03F5

1

/uni03F5

2

/uni03F5

3

Accumulated

noise

ξ

/uni03F5

1

GPES

Worker

K

=2

W

/uni03F5

1

/uni03F5

1

/uni03F5

1

⏞

Truncation window

Sample

NRESWorker

/uni03F5

2

+

/uni03F5

1

/uni03F5

2

+

/uni03F5

1

/uni03F5

2

+

/uni03F5

3

+

divides

K

τ

How different unbiased OES methods share noise?

No need to

accumulate

Accumulated

noise

length

W

=

ξ

=

Sample

/uni03F5

1

Sample

Reuse

/uni03F5

1

/uni03F5

2

Sample

Reuse

/uni03F5

/uni03F5

Reuse

/uni03F5

(b)

̸

function with lower sharpness and fewer local minima than before ( σ > 0 is a hyperparameter):

An unbiased gradient estimator of (2) is given by the likelihood ratio gradient estimator [16]: 1 σ 2 L ([ θ + ϵ ] × T ) ϵ . This estimator only requires the loss evaluation (hence is zeroth-order) but not an explicit computation of the gradient, thus being applicable in cases when the gradients are noninformative (chaotic or piecewise constant loss) or not directly computable (black-box loss). To reduce the variance, antithetic sampling is used and we call this estimator FullES (Algorithm 3 in the Appendix):

The term Full highlights that this estimator can only produce a gradient estimate after a full sequential unroll from t = 0 to T . We call such a full unroll an episode following the reinforcement learning terminology. FullES can be parallelized [14] by averaging N parallel gradient estimates using i.i.d. ϵ 's, but is not online and can result in substantial latency between gradient updates when T is large.

Truncated Evolution Strategies. To make FullES online, Metz et al. [3] take inspiration from truncated backpropagation through time ( TBPTT ) and propose the algorithm TES (see Algorithm 6 in the Appendix). Unlike the stateless estimator FullES , TES is stateful: TES starts from a saved state s (from the previous iteration) and draws a new ϵ i for antithetic unrolling. To make itself online, TES only unrolls for a truncation window of W steps for every gradient estimate, thus reducing the latency from O ( T ) to O ( W ) . Analytically,

Here, besides the Gaussian random variables ϵ i ∣ ∣ T/W i =1 iid ∼N ( 0 , σ 2 I d × d ) , the time step τ which TES starts from is also a random variable drawn from the uniform distribution τ ∼ Unif { 0 , W, . . . , T -W } 6 . It is worth noting that Vicol et al. [15] who also analyze online ES estimators do not take the view that the time step τ an online ES estimator starts from is a random variable; as such their analyses do not fully reflect the 'online' nature captured in our work. When multiple online ES Workers (e.g., TES workers) run in parallel, different workers will work at different i.i.d. time steps τ i (which we call step-unlocked workers) (see Figure 2(a)). We provide the pseudocode for creating step-unlocked workers and for general online ES learning in Algorithm 4 and 5 in the Appendix.

TES is a biased gradient estimator of (2) . Note that in (4), only the θ 's in the current lengthW truncation window receive antithetic perturbations, thus ignoring the impact of the earlier θ 's up to time step τ . Due to this bias, optimization using TES typically doesn't converge to optimal solutions.

Persistent Evolution Strategies. To resolve the bias of TES , Vicol et al. [15] recognize that TES samples a new noise in every truncation window and modifies the smoothing objective into:

They show that an unbiased gradient estimator of 5 is given by (see Algorithm 1):

with randomness in both { ϵ i } and τ . To eliminate the bias of TES , instead of multiplying only with the current epsilon ϵ τ /W +1 , PES multiplies it with the cumulative sum of all the different iid noise sampled so far (self. ξ in PESWorker). As we shall see in the next section, this accumulation of noise terms provably results in higher variance, making PES less desirable in practice. We contrast the noise sampling properties of our proposed methods with PES in Figure 1(b).

Hysteresis. When online gradient estimators are used in training, they often suffer from hysteresis , or history dependence, as a result of the parameters θ changing between adjacent unrolls. That is, the parameter value θ 0 that a worker uses in the current truncation window is not the same parameter θ -1 that was used in the previous window. This effect is often neglected [15], under an assumption that θ is updated slowly. To the best of our knowledge, [17] is the only work to analyze the convergence of an online gradient estimator under hysteresis. In the following theoretical analysis, we assume all online gradient estimates are computed without hysteresis in order to isolate the problem. However, in Section 5, we show empirically that even under the impact of hysteresis, our proposed online estimator NRES can outperform non-online methods (e.g., FullES ) which don't suffer from hysteresis.

3 A New Class of Unbiased Online Evolution Strategies Methods

As shown in Section 2, TES and PES both sample a new noise perturbation ϵ for every truncation window to produce gradient estimates. Here we note that the frequency of noise-sharing (new noise every truncation window of size W ) is fixed to the frequency of gradient estimates (a gradient estimate every truncation window of size W ). However, the former is a choice of the smoothing objective (5), while the latter is often a choice of how much gradient update latency the user can tolerate. In this section we break this coupling by introducing a general class of gradient estimators that encompass PES . We then analyze these estimators' variance to identify the one with the least variance.

Generalized Persistent Evolution Strategies (GPES). For a given fixed truncation window size W , we consider all noise-sharing periods K that are multiples of W , K = cW for c ∈ Z + , c ≤ T/W . K being a multiple of W ensures that within each truncation window, only a single ϵ is used. When

K = W , we recover the PES algorithm. However, when K is larger than W , the same noise will be used across adjacent truncation windows (Figure 2(b)). With a new noise sampled every K unroll steps, we define the K -smoothed loss objective as the function:

where r : ( Z + ) 2 → Z + is the modified remainder function such that r( x, y ) is the unique integer n ∈ [1 , y ] where x = qy + n for some integer q . This extra notation allows for the possibility that T is not divisible by K and the last noise ϵ ⌈ T/K ⌉ is used for only r( T, K ) < K steps.

We now give the analytic form of an unbiased gradient estimator of the resulting smoothed loss. 8 Lemma 1. An unbiased gradient estimator for the K -smoothed loss is given by

with randomness in τ and { ϵ i } ⌈ T/K ⌉ i =1 .

GPES K algorithm. Here, for the truncation window starting at step τ , the noise ϵ ⌊ τ /K ⌋ +1 is used as the antithetic perturbation to unroll the system. If τ is not divisible by K , then this noise has already been sampled at time step t = ⌊ τ /K ⌋ · K in an earlier truncation window. Therefore, to know what noise to apply at this truncation window, we need to remember the last used ϵ and update it when τ becomes divisble by K . We provide the algorithm for the GPES K gradient estimator in Algorithm 7 in the Appendix. Note that GPES K = W is the same as the PES algorithm.

Variance Characterization of GPES K . With this generalized class of gradient estimators GPES K , one might wonder how to choose the value of K . Since each estimator is an unbiased gradient estiamtor with respect to its smoothed objective, we compare the variance of these estimators as a function of K . To do this analytically, we make some simplifying assumptions:

Assumption 2. For a given θ ∈ R d and t ∈ [ T ] := { 1 , . . . , T } , there exists a set of vectors { g t i ∈ R d } t i =1 , such that for any { v i ∈ R d } t i =1 , the following equality holds:

Remark 3. This assumption is more general than the quadratic L t assumption made in [15] (explanation see Appendix D). Here one can roughly understand g t i as time step t 's smoothed loss's partial derivative with respect to the i -th application of θ . For notational convenience, we let g t := ∑ t i =1 g t i (roughly the total derivative of smoothed stept loss with respect to θ ) and g t K,j := ∑ min { t, K · j } i = K · ( j -1)+1 g t i for j ∈ { 1 , . . . , ⌈ t/K ⌉} (roughly the sum of partial derivatives of smoothed stept loss with respect to all θ 's in the j -th noise-sharing window of size K (the last window might be shorter)).

With this assumption in place, we first consider the case when W = 1 and K = cW = c for c ∈ [ T ] . In this case, the GPES K estimator can be simplified into the following form:

With this simplified form, we can now characterize the variance of the estimator GPES K = c ( θ ) . Since it's a random vector, we analytically derive its total variance (trace of covariance matrix) tr(Cov[GPES K = c ( θ )]) .

Theorem 5. When W = 1 and under Assumption 2, for integer c ∈ [ T ] ,

1

(PES)

4

16

64

256

1000

(NRES)

noise-sharing period

K

[1,

T

]

10 3

10 4

10 5

10 6

tr(Cov[GPES

K

(

)])

Learned Optimizer

T

=1000,

W

=1

10000

20000

30000

40000

50000

60000

70000

80000

To understand how the value of K = c changes the total variance, we notice that only the nonnegative third term in (9) depends on it. This term measures the pairwise squared distance between nonoverlapping partial sums g t c,j for all j . When c = T , for every t ∈ [ T ] , there is only a single such partial sum as ⌈ t/c ⌉ = 1 . In this case, this third term reduces to its smallest value of 0 . Thus:

Corollary 6. Under Assumption 2, when W = 1 , the gradient estimator GPES K = T ( θ ) has the smallest total variance among all { GPES K : K ∈ [ T ] } estimators.

Remark 7. To understand Corollary 6 intuitively, notice that at a given time step t (i.e., a length-1 truncation window), any GPES K = c gradient estimator ( c ∈ [ T ] ) aims to unbiasedly estimate the total derivative of the smoothed loss at this step with respect to θ , which we have denoted by g t . By applying a new Gaussian noise perturbation every c < T steps, the GPES K = c estimators indirectly estimate g t by first unbiasedly estimating the gradients inside each sizec noise-sharing window: { g t c,j } ⌈ t/c ⌉ j =1 and then summing up the result (notice g t = ∑ ⌈ t/c ⌉ j =1 g t c,j ). To obtain this extra (yet unused) information about the intermediate partial derivatives, these estimators require more randomness and thus suffer from a larger total variance than the GPES K = T estimator which directly estimates g t .

Experimental Verification of Corollary 6. We empirically verify Corollary 6 on a meta-training learned optimizer task ( T = 1000 ; see additional details in Section 5.2). Here we save a trajectory of θ i learned by PES ( i denotes training iteration) and compute the total variance of the estimated gradients (without hysteresis) by GPES with different values of K in Figure 3(a) ( W = 1 ). In agreement with theory, K = T has the lowest variance.

4 Noise-Reuse Evolution Strategies

NRES has lower variance than PES. As variance reduction is desirable in stochastic optimization [18], by Corollary 6, the gradient estimator GPES K = T is particularly attractive and can serve as a variance-reduced replacement for PES . When K = T , we only need to sample a single ϵ once at the beginning of an episode (when τ = 0 ) and reuse the same noise for the entirety of that episode before it resets. This removes the need to keep track of the cumulative applied noise ( ξ ) (Figure 2(b)), making the algorithm simpler and more memory efficient than PES . Due to its noise-reuse property, we name this gradient estimator GPES K = T the Noise-Reuse Evolution Strategies ( NRES ) (pseudocode in Algorithm 2). Concurrent with our work, Vicol [19] independently proposes a similar algorithm with different analyses. We discuss in detail how our work differs from [19] in Appendix B. Despite Theorem 5 assuming W = 1 , one can relax this assumption to any W that divides the horizon length T . By defining a 'mega' UCG whose single transition step is equivalent to W steps in the original UCG, we can apply Corollary 6 to this mega UCG and arrive at the following result.

Corollary 8. Under Assumption 2, when W divides T , the NRES gradient estimator has the smallest total variance among all GPES K = cW estimators c ∈ [ T/W ] .

NRES is a replacement for FullES. By sharing the same noise over the entire horizon, the smoothing objective of NRES is the same as FullES 's. Thus, we can think of NRES as the online counterpart to the offline algorithm FullES . Hence NRES can act as a drop-in replacement to FullES in UCGs. A single FullES worker runs 2 T unroll steps for each gradient estimate, while a single NRES runs only 2 W steps. Motivated by this, we compare the average of T/W i.i.d. NRES gradient estimates with 1 FullES gradient estimate as they require the same amount of compute.

NRES is more parallelizable than FullES. Because the T/W NRES gradient estimators are independent of each other, we can run them in parallel. Under perfect parallelization, the entire NRES gradient estimation would require O ( W ) time to complete. In contrast, the single FullES gradient estimate has to traverse the UCG from start to finish, thus requiring O ( T ) time. Hence, NRES is T/W times more parallelizable than FullES under the same compute budget (Figure 4(a)).

̂

̂

̂

• Returns

•

time (parallelization)

• Total variance lower than

(Theorem 9)

W

T

T

/

W

∑

i

=1

g

(

i

)

O

(

W

)

g

window length

W

/uni23DF

Gradient estimation of FullES vs NRES

under the same per-update budget of

unrolls

2

T

non-parallelizable

Worker

(

T

W

)

FullES

NRES

g

(

T

W

)

θ

±

/uni03F5

(2)

Episode length

T

•

time to return

O

(

T

)

g

unrolls in the

current update

using the same

perturbation

one episode

g

(1)

g

(2)

Worker

(2)

Worker

(1)

Worker

θ

±

/uni03F5

θ

±

/uni03F5

θ

±

/uni03F5

(1)

θ

±

/uni03F5

parallelizable

same starting time

(

T

W

)

g

(a)

(b)

T

= 1000

,

W

= 1

(

±

95%

total variance

tr(Cov[

·

])

iter.

j

W

T

T/W

∑

i

=1

NRES

i

(

θ

j

)

FullES(

θ

j

)

of

θ

j

(

×

10

4

)

1

1

.

47

±

0

.

38

864

.

34

±

385

.

01

2

15

.

74

±

4

.

90

513

.

50

±

74

.

71

3

4

.

28

±

0

.

78

201

.

38

±

30

.

18

4

3

.

20

±

0

.

50

684

.

85

±

86

.

90

5

4

.

47

±

0

.

88

181

.

06

±

24

.

97

6

4

.

42

±

0

.

68

288

.

61

±

46

.

77

7

5

.

33

±

1

.

08

448

.

30

±

62

.

44

8

6

.

62

±

0

.

89

154

.

86

±

28

.

21

̂

NRES can often have lower variance than FullES. We next compare the variance of the average of T/W i.i.d. NRES gradient estimates with the variance of 1 FullES gradient estimate:

Theorem 9. Under Assumption 2, for any W that divides T , if

then for iid { NRES i ( θ ) } T/W i =1 estimators, tr(Cov( 1 T/W ∑ T/W i =1 NRES i ( θ )) ≤ tr(Cov(FullES( θ )) .

Remark 10. To understand the inequality assumption in (10), we notice that it relates the sum of the squared 2 -norm of vectors { ∑ W · k t = W · ( k -1)+1 g t } T/W k =1 with the squared 2 -norm of their sum. When these vectors are pointing in similar directions, this inequality would hold (to see this intuitively, consider the more extreme case when all these vectors are exactly in the same direction). Because each term ∑ W · k t = W · ( k -1)+1 g t can be understood as the total derivative of the sum of smoothed losses in the k -th truncation window with respect to θ , we see that inequality (10) is satisfied when, roughly speaking, different truncation windows' gradient contributions are pointing in similar directions. This is often the case for real-world applications because if we can decrease the losses within a truncation window by changing the parameter θ , we likely will also decrease other truncation windows' losses. At a high-level, Theorem 9 shows that for many practical unrolled computation graphs, NRES is not only better than FullES due to its better parallelizability but also better due to its lower variance given the same computation budget.

Empirical Verification of Theorem 9. We empirically verify Theorem 9 in Figure 4(b) using the same set up of the meta-training learned optimizer task used in Figure 3(a). Here we compare the total variance of averaging T/W = 1000 i.i.d. NRES estimators versus using 1 FullES gradient estimator (same total amount of compute). We see that NRES has a significantly lower total variance than FullES while also allowing T/W = 1000 times wall-clock speed up due to its parallelizability.

5 Experiments

NRES is particularly suitable for optimization in UCGs in two scenarios: 1) when the loss surface exhibits extreme local sensitivity; 2) when automatic differentiation of the loss is not possible/gives noninformative (e.g., zero) gradients. In this section, we focus on three applications exhibiting these properties: a) learning Lorenz system's parameters (sensitive), b) meta-training learned optimizers (sensitive), and c) reinforcement learning (nondifferentiable), and show that NRES outperforms existing AD and ES methods for these applications. When comparing online gradient estimation methods, we keep the number of workers N used by all methods the same for a fair comparison. For the offline method FullES , we choose its number of workers to be W/T × the number of NRES workers on all tasks (in order to keep the same number of unroll steps per-update) except for the learned optimizer task in Section 5.2 where we show that NRES can solve the task faster while using much fewer per-update steps than FullES .

̂

̂

̂

(a)

0

50

100

150

200

250

300

wall-clock time (seconds)

10 2

4×10 1

6×10 1

2×10 2

3×10 2

loss

5 random seeds average (95% confidence interval)

FullES

N

=10

TES

W

=100,

N

=200

PES

W

=100,

N

=200

NRES

W

=100,

N

=200 (ours)

PES

W

=100,

N

=20000

converged value

NRES

W

=100,

N

=20000

converged value

T

=2000

(b)

traininglossL([θ]xT)

LorenzsystemT=2ooo

θ =(In(r),In(a))

50

250 450

optimizationtrajectories(θ;)bydifferentGPESk

N=200,W=100,α=0.04,SGDlearningrate:1e-5

3.2

0o

K=W

(PES)

K=4W

K=16W

K=T

(NRES)

iteration

θgt

i

3.0

0

(e)ul

2.8

2.6

2.4

2.2

1200

3.23.43.6

3.23.43.6

3.23.43.6

3.23.43.6

3.23.43.6

In(r)

5.1 Learning dynamical system parameters

In this application, we consider learning the parameters of a Lorenz system, a canonical chaotic dynamical system. Here the state s t = ( x t , y t , z t ) ∈ R 3 is unrolled with two learnable parameters a, r 9 with the discretized transitions ( dt = 0 . 005 ) starting at s 0 = ( x 0 , y 0 , z 0 ) = (1 . 2 , 1 . 3 , 1 . 6) :

Due to the positive constraint on r > 0 and a > 0 , we parameterize them as θ = (ln( r ) , ln( a )) ∈ R 2 and exponentiate the values in each application. We assume we observe the ground truth z -coordinate z gt t for t ∈ [ T ] , T = 2000 steps unrolled by the default parameters ( r gt , a gt ) = (28 , 10) . For each step t , we measure the squared loss L s t ( s t ) := ( z t -z gt t ) 2 . Our goal is to recover the ground truth parameters θ gt = (ln(28) , ln(10)) by optimizing the average loss over all time steps using vanilla SGD. We first visualize the training loss surface in the left panel of Figure 5(a) (also see Figure 8 in the Appendix) and notice that it has extreme sensitivity to small changes in the parameter θ .

To illustrate the superior variance of NRES over other GPES K estimators, we plot in the right panels of Figure 5(a) the optimization trajectory of θ using gradient estimator GPES K with different values of K under the same SGD learning rate. We see that NRES 's trajectory exhibits the least amount of oscillation due to its lowest variance. In contrast, we notice that PES 's trajectory is highly unstable, thus requiring a smaller learning rate than NRES to achieve a possibly slower convergence. Hence, we take extra care in tuning each method's constant learning rate and additionally allow PES to have a decay schedule. We plot the convergence of different ES gradient estimators in wall-clock time using the same hardware in Figure 5(b). (We additionally compare against automatic differentiation methods in Figure 9 in the Appendix; they all perform worse than the ES methods shown here.)

In terms of the result, we see that NRES outperforms 1) TES , as NRES is unbiased and can better capture long-term dependencies; 2) PES , as NRES has provably lower variance, which aids convergence in stochastic optimization; 3) FullES , as NRES can produce more gradient updates in the same amount of wall clock time than FullES (with parallelization, each NRES update takes O ( W ) time instead of FullES 's O ( T ) time). Additionally, we plot the asymptotically converged loss value when we train with a significantly larger number of particles ( N = 20000 ) for PES and NRES . We see that by only using N = 200 particles, NRES can already converge around its asymptotic limit, while PES is still far from reaching its limit within our experiment time.

5.2 Meta-training learned optimizers

In this application [3], the meta-parameters θ of a learned optimizer control the gradient-based updates of an inner model's parameters. The inner state s t is the optimizer state which consists of both the inner model's parameters and its current gradient momentum statistics. The transition function f t computes an additive update vector to the inner parameters using θ and a random training batch and outputs the next optimizer state s t +1 . Each time step t 's meta-loss L s t evaluates the updated inner parameters' generalization performance using a sampled validation batch.

We consider meta-training the learned optimizer model given in [3] ( d = 1762 ) to optimize a 3layer MLP on the Fashion MNIST dataset for T = 1000 steps. (We show results on the same task

8000

0

1000

2000

3000

4000

5000

6000

7000

8000

wall-clock time (seconds)

10 0

6×10 1

2×10 0

non-smoothed training loss

3456.6

691.7

6313.7

5 random seeds average (95% confidence interval)

FullES

N

=10

TES

W

=1,

N

=100

PES

W

=1,

N

=100

NRES

W

=1,

N

=100 (ours)

FullES

N

=10

TES

W

=1,

N

=100

PES

W

=1,

N

=100

NRES

W

=1,

N

=100 (ours)

Learned Optimizer

T

=1000

(a)

1000

2000

3000

4000

5000

6000

7000

wall-clock time (seconds)

10 0

5×10 1

6×10 1

7×10 1

8×10 1

9×10 1

non-smoothed training loss

5 random seeds average (95% confidence interval)

PES (GPES

K

=1)

GPES

K

=4

GPES

K

=16

GPES

K

=64

GPES

K

=256

NRES (GPES

K

=1000)

PES (GPES

K

=1)

GPES

K

=4

GPES

K

=16

GPES

K

=64

GPES

K

=256

NRES (GPES

K

=1000)

Learned Optimizer

T

=1000,

W

=1,

N

=100

(b)

with higher-dimension and longer horizon in Appendix E.1.2.) This task is used in the training task distribution of the state of the art learned optimizer VeLO [20] 10 . The loss surface for this problem has high sharpness and many suboptimal minima as previously shown in Figure 1(a). We meta-train with Adam using different gradient estimation methods with the same hardware and tune each gradient estimation method's meta learning rate individually. Because AD methods all perform worse than the ES methods, we defer their results to Figure 10 in the Appendix and only plot the convergence of the ES methods in wall-clock time in Figure 6(a).

Here we see that NRES reaches the lowest loss value in the same amount of time. In fact, PES and FullES would require 5 and 9 × (respectively) longer than NRES to reach a loss NRES reaches early on during its training, while TES couldn't even reach that loss within our experiment time. It is worth noting that, for this task, NRES only require 2 · N · W = 200 unrolls to produce an unbiased, low-variance gradient estimate, which is even smaller than the length of a single episode ( T = 1000) . In addition, we situate NRES 's performance within our proposed class of GPES K estimators in Figure 6(b). In accordance with Corollary 8, NRES converges fastest due to its reduced variance.

5.3 Reinforcement Learning

It has been shown that ES is a scalable alternative to policy gradient and value function methods for solving reinforcement learning tasks [14]. In this application, we learn a linear policy 11 (following [21]) using different ES methods on the Mujoco [22] Swimmer ( d = 16 ) and Half Cheetah task ( d = 102 ). We minimize the average of negative per-step rewards over the horizon length of T = 1000 , which is equivalent to maximizing the undiscounted sum rewards. Unlike [14, 21, 15], we don't use additional heuristic tricks such as 1) rank conversion of rewards, 2) scaling by loss standard deviation, or 3) state normalization. Instead, we aim to compare the pure performance of different ES methods assuming perfect parallel implementations. To do this, we measure a method's performance as a function of the number of sequential environment steps it used. Sequential environment steps are steps that have to happen one after another (e.g., the environment steps within the same truncation window). However, steps that are parallelizable don't count additionally in the sequential steps. Hence, the wall-clock time under perfect parallel implementation is linear with respect to the number of sequential environment steps used. As all the methods we compare are iterative update methods, we additionally require that each method use the same number of environment steps per update when measuring each method's required number of sequential steps to solve a task. We tune the SGD learning rate individually for each method and plot their total rewards progression on both Mujoco tasks in Figure 7.

Here we see that TES fails to solve both tasks due to the short horizon bias [23], making it unable to capture the long term dependencies necessary to solve the tasks. On the other hand, PES , despite being unbiased, suffers from high variance, making it take longer (or unable in the case of Half Cheetah) to solve the task than NRES . As for FullES , despite using the same amount of compute per gradient update as NRES , it's much less parallelizable as discussed in Section 4 - it takes much longer time ( 10 × and 60 × ) than NRES assuming perfect parallelization. In addition to the number

10 6

10 3

10 4

# of sequential environment steps

0

100

200

300

400

undiscounted sum rewards

2400

25000

10300

5 random seeds average (95% confidence interval)

FullES

N

=3

TES

W

=100,

N

=30

PES

W

=100,

N

=30

NRES

W

=100,

N

=30 (ours)

task solving threshold

Swimmer,

T

=1000

(a)

10 3

10 4

10 5

# of sequential environment steps

1000

0

1000

2000

3000

4000

undiscounted sum rewards

628000

10200

5 random seeds average (95% confidence interval)

FullES

N

=6

TES

W

=20,

N

=300

PES

W

=20,

N

=300

NRES

W

=20,

N

=300 (ours)

task solving threshold

Half Cheetah,

T

=1000

(b)

of sequential steps, we additionally show the total number of environment steps used by each method in Table 3 in the Appendix NRES also uses the least total number of steps to solve both tasks, making it the most sample efficient.

6 Additional Related Work

Beyond the most related work in Section 2, in this section, we further position NRES relative to existing zeroth-order gradient estimation methods. We also provide additional related work on automatic differentiation (AD) methods for unrolled computation graphs in Appendix B.

Zeroth-Order Gradient Estimators. In this work, we focus on zeroth-order methods that can estimate continuous parameters' gradients to be plugged into any first-order optimizers, unlike other zeroth-order optimization methods such as Bayesian Optimization [24], random search [25], or Trust Region methods [26, 27]. We also don't compare against policy gradient methods [9], because they assume internal stochasticity of the unrolling dynamics, which may not hold for deterministic policy learning [e.g., 22]. Within the space of evolution strategies methods, many works have focused on improving the vanilla ES method's variance by changing the perturbation distribution [28-31], considering the covariance structure [32], and using control variates [33]. However, these works do not consider the unrolled structure of UCGs and are offline methods. In contrast, we reduce the variance by incorporating this unrolled aspect through online estimation and noise-reuse. As the aforementioned variance reduction methods work orthogonally to NRES , it is conceivable that these techniques can be used in conjunction with NRES to further reduce the variance.

7 Discussion, Limitations, and Future Work

In this work, we improve online evolution strategies for unbiased gradient estimation in unrolled computation graphs by analyzing the best noise-sharing strategies. By generalizing an existing unbiased method, Persistent Evolution Strategies, to a broader class, we analytically and empirically identify the best estimator with the smallest variance and name this method Noise-Reuse Evolution Strategies ( NRES ). We demonstrate the convergence benefits of NRES over other automatic differentiation and evolution strategies methods on a variety of applications.

Limitations. As NRES is both an online method and an ES method, it naturally inherits some limitations shared by all methods of these two classes, such as hysteresis and variance's linear dependence on the dimension d . We provide a detailed discussion of these limitations in Appendix F.

Future Work. There are some natural open questions: 1) choosing a better sampling distribution for NRES . Currently the isotropic Gaussian's variance σ 2 is tuned as a hyperparameter. Whether there are better ways to leverage the sequential structure in unrolled computation graphs to automate the selection of this distribution is an open question. 2) Incorporating hysteresis. Our analysis assumes no hysteresis in the gradient estimates and we haven't observed much impact of it in our experiments. However, understanding when and how to correct for hysteresis is an interesting direction.

Acknowledgments

We thank Kevin Kuo, Jingnan Ye, Tian Li, and the anonymous reviewers for their helpful feedback.

References

Sepp Hochreiter and J¨ urgen Schmidhuber. Long short-term memory. Neural Computation , 9 (8):1735-1780, 1997.

Kyunghyun Cho, Bart Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP , 2014.

Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning , 2019.

James Harrison, Luke Metz, and Jascha Sohl-Dickstein. A closer look at learned optimization: Stability, robustness, and inductive biases. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In International Conference on Machine Learning , 2015.

Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning , 2017.

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4750-4759, 2022.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems , 1999.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning , 2015.

Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research , 18:1-43, 2018.

Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. PIPPS: Flexible modelbased policy search robust to the curse of chaos. In International Conference on Machine Learning , 2018.

Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman. Gradients are not all you need. arXiv preprint arXiv:2111.05803 , 2021.

Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864 , 2017.

Paul Vicol, Luke Metz, and Jascha Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In International Conference on Machine Learning , 2021.

Peter W Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM , 33(10):75-84, 1990.

Pierre-Yves Mass´ e and Yann Ollivier. Convergence of online adaptive and recurrent optimization algorithms. arXiv preprint arXiv:2005.05645 , 2020.

Chong Wang, Xi Chen, Alexander J Smola, and Eric P Xing. Variance reduction for stochastic gradient optimization. Advances in neural information processing systems , 26, 2013.

Paul Vicol. Low-variance gradient estimation in unrolled computation graphs with es-single. In International Conference on Machine Learning , 2023.

Luke Metz, James Harrison, C Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, et al. Velo: Training versatile learned optimizers by scaling up. arXiv preprint arXiv:2211.09760 , 2022.

Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. In Advances in Neural Information Processing Systems , 2018.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems , 2012.

Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. In International Conference on Learning Representations , 2018.

Peter I Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811 , 2018.

James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research , 13(2), 2012.

Alvaro Maggiar, Andreas Wachter, Irina S Dolinskaya, and Jeremy Staum. A derivative-free trust-region algorithm for the optimization of functions smoothed via gaussian convolution using adaptive multiple importance sampling. SIAM Journal on Optimization , 28(2):1478-1507, 2018.

Guoqing Liu, Li Zhao, Feidiao Yang, Jiang Bian, Tao Qin, Nenghai Yu, and Tie-Yan Liu. Trust region evolution strategies. In AAAI Conference on Artificial Intelligence , 2019.

Krzysztof Choromanski, Mark Rowland, Vikas Sindhwani, Richard Turner, and Adrian Weller. Structured evolution with compact architectures for scalable policy optimization. In International Conference on Machine Learning , 2018.

Niru Maheswaranathan, Luke Metz, George Tucker, Dami Choi, and Jascha Sohl-Dickstein. Guided evolutionary strategies: augmenting random search with surrogate gradients. In International Conference on Machine Learning , 2019.

Alexandru Agapie. Spherical distributions used in evolutionary algorithms. Mathematics , 9 (23), 2021. ISSN 2227-7390.

Katelyn Gao and Ozan Sener. Generalizing Gaussian smoothing for random search. In International Conference on Machine Learning , 2022.

Nikolaus Hansen. The cma evolution strategy: A tutorial. arXiv preprint arXiv:1604.00772 , 2016.

Yunhao Tang, Krzysztof Choromanski, and Alp Kucukelbir. Variance reduction for evolution strategies via structured control variates. In International Conference on Artificial Intelligence and Statistics , 2020.

Mart´ ın Abadi. Tensorflow: learning functions at scale. In Proceedings of the 21st ACM SIGPLAN International Conference on Functional Programming , pages 1-1, 2016.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems , 2019.

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 , 2016.

Corentin Tallec and Yann Ollivier. Unbiasing truncated backpropagation through time. arXiv preprint arXiv:1705.08209 , 2017.

Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. Advances in Neural Information Processing Systems , 2017.

Ronald J Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation , 1(2):270-280, 1989.

David Silver, Anirudh Goyal, Ivo Danihelka, Matteo Hessel, and Hado van Hasselt. Learning by directional gradient descent. In International Conference on Learning Representations , 2021.

Corentin Tallec and Yann Ollivier. Unbiased online recurrent optimization. arXiv preprint arXiv:1702.05043 , 2017.

Asier Mujika, Florian Meier, and Angelika Steger. Approximating real-time recurrent learning with random kronecker factors. Advances in Neural Information Processing Systems , 31, 2018.

Frederik Benzing, Marcelo Matheus Gauy, Asier Mujika, Anders Martinsson, and Angelika Steger. Optimal kronecker-sum approximation of real time recurrent learning. In International Conference on Machine Learning , 2019.

Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Luke Metz, C Daniel Freeman, James Harrison, Niru Maheswaranathan, and Jascha SohlDickstein. Practical tradeoffs between memory, compute, and performance in learned optimizers. In Conference on Lifelong Learning Agents , 2022.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe study considers numerically estimating gradients for online (reinforcement) learning.\nThe parameters are perturbed and from the performance of the perturbed models the gradient is estimated.\nIn an online setting, the proposed method decouples the number of steps between gradient estimates (which are then used by a first order gradient based optimiser) and the number of steps between changing the perturbation.\nTwo aspects are relevant in the paper: The variance of the gradient estimate and the ability to parallelise the methods.\n\nSTRENGTHS:\nI could not find anything that I would call wrong in the manuscript.\n\nWEAKNESSES:\nAllmost all research is to a certain degree incremental. The main problem of the research presented in the paper that is that the increment is rather small, it lacks originality and novel „surprising“ insights. \n\nThe problem setting is not new.\nThe proposed algorithm, as properly stated in the manuscript, can be viewed as a slight generalisation of „Persistent evolution strategies“ [13].\nThe canonical way to apply a Monte Carlo method such as ES to non-episodic RL tasks is to split the state-action-reward sequence into „pseudo-episodes“ . This is also done/suggested in other other non-ES RL algorithms, e..g., in \tProximal Policy Optimization. This is also sometimes referred to as truncation. \nThe problem of how many (pseudo-)episodes are needed to evaluate a perturbation to get a reliable signal to be exploited by ES for RL has also been studied, for example see \nHeidrich-Meisner and Igel. Hoeffding and Bernstein Races for Selecting Policies in Evolutionary Direct Policy Search. ICML 2009\n\nTwo aspects are relevant in the paper: The variance of the gradient estimate and the ability to parallelise the methods. The first one leads to a non-surprising result, which was simultaneously discovered in another work, properly cited by the authors: [19].\nIn my evaluation, the patalllization, while being useful in practice, is rather technicality.\n\nThe discussion the (parallel) runtime is partly not convincing.\nI understand that \n1. FullES takes O(T) time, it sees the states from s^+_0 to s^+_T (and s^-_0 to s^-_T)\n2. calling Algorithm 2 takes O(W) time, it sees the states s^+_tau to s^+_tau+W (and s^-_tau to s^-_tau+W)\n3. T/W independent calls of Algorithm 2 can be fully parallelised, so on T/W nodes taking the average over T/W independent calls starting from the same state s+ / s- also takes O(W) time\n(Please correct me if I am wrong).\n\nSo I think „Under perfect parallelization, the entire  NRES gradient estimation would require O(W) time to complete. In contrast, the single FullES  gradient estimate has to traverse the UCG from start to finish, thus requiring O(T) time. Hence, NRES is T /W times more parallelizable than FullES with the same compute budget.“ is a bit misleading. Because a single FullES  gradient estimate sees all steps from 0 to T, while Algorithm 2 looks at several „parallel“ trajectories starting from the same state running for W steps. I think this is qualitatively different. In the RL setting, you cannot in general not replace   information collected over a time period with information collected from an ensemble where the ensemble members start from the same state. Think of an RL task where the rewards are always zero in the first 0.99*T steps.  \n\nThe „Instead, we aim to compare the pure performance of different ES methods   assuming perfect parallel implementations. To do this, we measure a method’s performance as a  function of the number of sequential environment steps it used. Sequential environment steps are    steps that have to happen one after another (e.g., the environment steps within the same truncation window). However, steps that are parallelizable don’t count additionally in the sequential steps.“ \nThis means that global random search, which evaluates 10^50 independent random policies has only a cost of one?\nThis would then be the ultimate baseline method, outperforming all proposed algorithms. \n\nThe manuscript lacks clarity. For example, some statements about „chaotic loss surfaces“ are confusing. Being chaotic is a well defined mathematical property.\nIt starts to make sense in the experiments. The Lorenz system is chaotic, which inmates the first place refers to perturbations of the system state.\nThe experiments always start from the same initial state. Optimised are the coefficients of the Lorenz system. What is the exact definition of a  „chaotic loss surfaces“? Where is the proof that the loss surface is really chaotic when changing r and a (citation missing). \n\nA comparison with an evolution strategy that uses the perturbations also for the update, such as (1+1) CMA-ES with restricted covariance matrix) would have been interesting.\n\nHyperparameter tuning seems to be crucial. The description in the appendix is appreciated. \nThe Lorenz training suffers from instabilities. There are countermeasures, like gradient clipping.\nOne could also move to an optimiser that decouple gradient magnitude and update step length, for example Adam. Adam is used later, but for the \nLorenz experiments the manuscript sticks to SGD. How are the results with Adam (and corresponding learning rate tuning)?\nThis may be important because the experiments lead to to strong statements „AD methods perform worse than ES methods“ (Appendix) and one has to rule out that the problem is not the optimiser but really the AD as such.\n\n\n\nMinor:\n\n* Instead of talking about a trick and adding a reference, simply state that you use \\nabla_theta E_x~p(x|theta) [ f(x) ] = E_x~p [ f(x)  \\nabla_theta \\ln p(x|theta), that is clearer (referring to this as a trick makes me cringe).\n\n* The embedding in the existing literature is weak. One may not need no go back to Rechenberg and Schwefel when talking about evolution strategies (although it does not harm), but citing [12] as the only source for evolution strategies for RL is in my evaluation not OK.\n* The first epsion in line 73should be in bold font.\n* The hyperparameters are relevant, the main paper should point out to the appendix where the hyperparameters selection is discussed more clearly.\n\nQUESTIONS:\n* What is the exact definition of a  „chaotic loss surfaces“? Where is the proof that the Lorenz „loss surface“ is really chaotic when changing r and a?\n* Under the used „sequential environment steps“ measure, would a  global random search algorithm that  which evaluates 10^50 independent random policies only have a cost of one? Would this be the ultimate baseline method, outperforming all proposed algorithms?\n\nLIMITATIONS:\nOK", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

comments: SUMMARY:
This paper generalizes PES based on noise-reuse, generating a more general class of unbiased online ES gradient estimators. The authors analytically characterize the variance of the estimators and identify the lowest-variance estimator named Noise-Reuse Evolution Strategies (NRES). Experiments on learning dynamical systems, meta-training learned optimizers, and reinforcement learning show that NRES results in faster convergence than existing AD and ES methods in terms of wall-clock time and number of unroll steps.

STRENGTHS:
1. The paper proposes a general framework GPES by generalizing the existing PES algorithm based on noise-reuse.
2. The paper gives a theoretical analysis of unbiasedness and variance for GPES, and identifies the lowest-variance gradient estimator under this setting named NRES, which always reuses noise.
3. The paper proves that under some reasonable conditions, NRES has lower variance than FullES.

WEAKNESSES:
1. The proposed GPES is a simple and direct generalization of the existing PES. The novelty is not very strong.

2. The experiments need to be improved, e.g., adding the empirical comparison on longer sequences and higher dimension problems.

3. Why GPES can be better than PES? An intuitive explanation is needed.

Minor issue: Step-unlocked is used without explanation.

QUESTIONS:
1. In Theorem 8, it's said that the required assumption often holds in real-world applications. Can you give more explanation and evidences?

2. In the experiments, the x-axis of figures is measured by wall-clock time. Why not use the number of iterations?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a method for optimizing unrolled computation graphs (e.g. recurrent networks, etc.). When using ES to optimize a computation graph, the graph must be fully rolled out. Recent methods (PES) have examined using a truncated window to optimize, so that optimization can occur without a full unroll. PES aims to unbias the estimator by accumulating truncation noise during an online rollout, however, it produces high variance. \n\nThis paper presents a generalization to PES, GPES, that decouples the frequency of noise injection and gradient estimation. A specific instantiation, named NRES, samples the noise only once per episode. This reduces variance as it removes the need for noise accumulation. \n\nExperiments are presented on a two-parameter chaotic Lorenz system, where NRES outperforms previous work. An RNN-based meta-learning setup is also evaluated wherein the transition function defines an update over inner parameters, along with a reinforcement learning setup to solve Mujoco tasks.\n\nSTRENGTHS:\nThis paper provides an extremely thorough investigation of their proposed framework. The framework includes a generalization of previous work, along with a theoretically-justified instantiation with desirable properties on unbiased estimation and low variance. This is an original idea that is promising. The quality and clarity of the writing is excellent. The work sets up for additional work in investigating the properties of GPES methods, including variations on the proposed NRES.\n\nWEAKNESSES:\nThe final experiments on meta-learning and reinforcement learning are lacking detail. A more precise experimental setup would strengthen the argument of the paper. In addition, comparisons to non-ES based methods would give more context on how such methods compare to other solutions in the field. The experiments present wall-clock time as the unified axis, but discussion in the paper mentions total computation budget -- a figuring comparing the methods using some consistent measure of computation would strengthen the paper.\n\nQUESTIONS:\nHow does NRES compare to FullES in terms of pure computational budget? What are the scaling patterns of additional parallelization?\nHow do non-ES methods perform on the Mujoco tasks?\n\nAdditionally, ablations on hyperparameters such as the noise variance and # of workers would grant insight into the nuance of the method.\n\nLIMITATIONS:\nLimitations are discussed in section 7 -- they largely relate to ES in general.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
In this paper, the author(s) extended the well-known Persistent Evolution Strategies via noise-reuse and proposed an improved version with reduced variance. The main contribution of this paper is to provide detailed mathematical proof to validate their claim.

STRENGTHS:
Considering this significant contribution to the evolution strategies and other related ML communities, I personally suggest accepting this high-quality research paper.

In this paper, the author(s) also discussed the main limits of their method. These discussions about the possible limitations (hysteresis and complexity) are highly encouraged for academic research because this can better reflect the whole state of their method.

Furthermore, the author(s) pointed out that “whether there are better ways to leverage the sequential structure” beyond the isotropic Gaussian distribution. I agree that this is an interesting open question (worth to be investigated).

WEAKNESSES:
Some notes are given in order to further improve this paper:
For Section 5.3 Reinforcement Learning, the removal of “additional heuristic tricks” can help us focus on the underlying mechanism, which is very critical for algorithmic understanding, no matter from a practical or theoretical perspective. We often prefer general-purpose design principles, though these “additional heuristic tricks” sometimes work well in some cases. Given that only the linear policy was used in experiments, it is highly expected to include also the more challenging non-linear policy with higher dimensions.

QUESTIONS:
see above

LIMITATIONS:
No comments here.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online evaluation strategies for unrolled computation graphs. Especially, the authors 1). propose a general class of unbiased online evolution strategies that generalizes Persistent Evolution Strategies (PES), named Generalized Persistent Evolution Strategies (GPES). The key idea is to share noise across truncation windows instead of sampling every round. 2). characterize the variance and variance reduction properties of their strategies. 3). study a special case of the general class strategies, named Noise Reuse Evolution Strategie (NRES), show the variance advantage of NRES over other estimators. 4). experimentally show the advantages of NRES across a variety of applications, including learning dynamical systems, meta-training learned optimizers, and reinforcement learning.\n\nSTRENGTHS:\n- The idea of reusing noise to reduce variance is interesting and a bit counterintuitive. For first order method, reusing noise always lead to a larger variance (If the noise is reused, averaging gradients cannot reduce the variance of the noise). However, it seems for evolution strategy, such a simple method could significantly reduce variance and improve the performance.\n- Experimental results are comparable to the previous methods and show a remarkable improvement.  \n- The paper is well writen. The proof in the appendix is well organized and seems correct.\n\nWEAKNESSES:\n- No comparision between NRES and first order method.\n- The authors did not explain the some parameter setting in the experiments. Especially, in Figure 5 and 6, why choose a different N when comparing online ES and FullES?\n- For online ES, the algorithm update $\\theta$ every truncation window, thus the loss will never be in the form of (7). In this case, it seems NRES is still biased.\n\nQUESTIONS:\nGiven a time $T$, FullES returns one gradient g=[L([\\theta+\\epsilon]_{\\times T})-L([\\theta-\\epsilon]_{\\times T})]{\\epsilon}/{2\\sigma^2}, and NRES (Algorithm 2) returns $T/W$ gradients $g_1,\\dots, g_{T/W}$, where g_k=[\\sum_{t=Wk+1}^{Wk+W}L_t([\\theta+\\epsilon]_{\\times t})-L([\\theta-\\epsilon]_{\\times t})]{\\epsilon}/{2\\sigma^2 W}. By definition, there is $g=W/T \\sum_{k=1}^{T/W} g_k$. In this case, if we choose learning rate $\\eta_{FullES} = \\eta_{NRES}T/W$, there should be no difference between FullES and NRES. So why is there such a big improvement in the experiments?\n\nOne minor typo: Algorithm 6 line 10: not $L_t^s$, should be $L_{self_.\\tau +1}^s$.\n\nLIMITATIONS:\nTheoretical work. No limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}