SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anastasia Koloskova* EPFL, Switzerland
Ryan McKenna  
Google Research
Zachary Charles Google Research
Keith Rush Google Research
Brendan McMahan Google Research
We study gradient descent under linearly correlated noise. Our work is motivated by recent practical methods for optimization with differential privacy (DP), such as DP-FTRL, which achieve strong performance in settings where privacy amplification techniques are infeasible (such as in federated learning). These methods inject privacy noise through a matrix factorization mechanism, making the noise linearly correlated over iterations. We propose a simplified setting that distills key facets of these methods and isolates the impact of linearly correlated noise. We analyze the behavior of gradient descent in this setting, for both convex and non-convex functions. Our analysis is demonstrably tighter than prior work and recovers multiple important special cases exactly (including anti-correlated perturbed gradient descent). We use our results to develop new, effective matrix factorizations for differentially private optimization, and highlight the benefits of these factorizations theoretically and empirically.
Differential privacy (DP) is a critical framework for designing algorithms with provable statistical privacy guarantees. DP stochastic gradient descent (DP-SGD, Abadi et al. [1]) is particularly important for enabling private empirical risk minimization (ERM) of machine learning models. Many works have analyzed the convergence behavior of DP ERM methods, including DP-SGD [5, 16, 48, 8]. However, obtaining good privacy/utility trade-offs with DP-SGD can require excessively large batch sizes or privacy amplification techniques such as subsampling [4, 5, 55] and shuffling [15, 16]. In some applications, including cross-device federated learning, limited and device-controlled client availability can make sampling or shuffling infeasible [21]. Even outside of such applications, many implementations of DP-SGD do not properly use the Poisson subsampling scheme analyzed by Abadi et al. [1] for amplification, and instead use a single fixed permutation of the dataset [7].
Kairouz et al. [20] propose an alternative method, DP-FTRL, which can attain good privacy/utility trade-offs without amplification. Their key insight is that for SGD-style algorithms, the variance on prefix sums  $\mathbf{g}_0 + \dots +\mathbf{g}_t$ $t\in \{1,\ldots ,T\}$  of gradients  $\mathbf{g}_j$  is more important than the variance on individual gradients. By adding carefully tailored noise that is linearly correlated over iterations to the gradients, one can reduce the error on the prefix sums, at the cost of increased error on the individual gradients, for a fixed privacy budget. The DP-FTRL mechanism is competitive with or better than DP-SGD, even without relying on privacy amplification, and enabled McMahan and Thakurta [31] to train the first differentially private machine learning model on user data in a production setting.
*Work performed while doing an internship at Google Research. Correspondence to: Anastasia Koloskova <anastasia.koloskova@epfl.ch>, Ryan McKenna <mckennar@gmail.com>.
Figure 1: Two-stage MF-DP-FTRL workflow proposed by Denisov et al. [10]. The user selects a workload matrix  $\mathbf{A}$  representing a desired first-order optimization method. Offline, the user finds a factorization  $\mathbf{BC} = \mathbf{A}$ , using an objective that balances ERM performance (as a function of  $\mathbf{B}$ ) and privacy (as a function of  $\mathbf{C}$ ). The user applies  $\mathbf{A}$  to a downstream ERM task, but with linearly correlated additive noise governed by  $\mathbf{B}$ .
Denisov et al. [10], Choquette-Choo et al. [7] develop a refinement of DP-FTRL, MF-DP-FTRL, by formulating and solving an offline matrix factorization problem to find the "optimal" correlated noise structure under DP constraints. That is, for a fixed privacy level, they aim to find correlated noise structures that lead to improved optimization. A simplified diagram of their workflow is given in Fig. 1. However, (as we detail in Section 2) their offline factorization objective is based on an online convergence bound that is loose. This raises questions about whether there are factorization objectives that better capture convergence behavior of gradient descent algorithms with correlated noise.
In this paper we study this class of mechanisms more closely and provide a detailed analysis of linearly correlated noise from an optimization point of view. Our main contributions are as follows:
- We propose a novel stochastic optimization problem that extracts key facets of methods like (MF-)DP-FTRL, and which isolates the effects of linearly correlated noise on optimization.
- We derive convergence rates for gradient descent on smooth convex and non-convex functions in such settings that showcase the effect of linearly correlated noise and recover tight convergence rates in notable special cases. We use a novel proof technique that may be of independent interest.
- We use this theory to design a new objective for the offline matrix factorization workflow in Fig. 1. We show that solving this objective leads to MF-DP-FTRL mechanisms with improved convergence properties. We validate the mechanism empirically on a variety of datasets and tasks, matching or outperforming prior methods.
Matrix mechanisms for differential privacy. Our work is closely related to differentially private optimization using matrix mechanisms [26]. Historically, such mechanisms were applied to linear statistical queries [25, 29, 14, 18]. Denisov et al. [10] and Choquette-Choo et al. [7] extended these mechanisms to the adaptive streaming setting, allowing their application to optimization with DP. Denisov et al. [10] show that this framework (MF-DP-FTRL) subsumes and improves the DP-FTRL algorithm [20]. Both DP-FTRL and MF-DP-FTRL improve privacy guarantees relative to DP-SGD [1] without amplification, and can be combined with techniques such as momentum for improved utility [46]. The aforementioned work focuses on methods for computing factorizations, privacy properties, and empirics. Our work studies the analytic relationship between the correlated noise induced by the MF-DP-FTRL framework and the downstream effect on optimization performance.
SGD with correlated noise. Stochastic noise in optimization arises in a variety of ways, including mini-batching [9] and explicit noise injection [11, 54, 19]. While most analyses of SGD assume this noise is independent across iterates, some work considers correlated noise. For example, shuffle SGD involves correlated noise due to sampling without replacement [33, 53]. Lucchi et al. [27]
use correlated Brownian motion to improve SGD's ability to explore the loss landscape. Recently, Orvieto et al. [37, 38] investigated anti-correlated noise as a way to impose regularization and improve generalization. We consider a linearly correlated noise model, and analyze its impact on SGD's convergence to critical points.
SGD with biased noise. Many algorithms can be viewed as SGD with structured but potentially biased noise, including SGD with (biased) compression [44, 17], delayed SGD [28, 12], local SGD [42], federated learning methods [22, 52, 34, 36], decentralized optimization methods [50, 23], and many others. Convergence analyses for such methods often use techniques like perturbed iterate analysis [28]. Correlated gradient noise also biases the gradient updates. However, as we show in Section 4, directly applying such techniques to linearly correlated noise does not lead to tight convergence guarantees.
In this work, we focus on an empirical risk minimization (ERM) problem of the form
$$
\min  _ {\mathbf {x} \in \mathbb {R} ^ {d}} \left[ f (\mathbf {x}) = \frac {1}{n} \sum_ {i = 1} ^ {n} l (\mathbf {x}, \xi_ {i}) \right], \tag {1}
$$
where  $l(\mathbf{x}, \xi_i)$  is the loss of a model  $\mathbf{x}$  on a data point  $\xi_i$ , and  $n$  is the training set size. We would like to solve (1) while guaranteeing some form of privacy for the training set. We focus on differential privacy (DP, [13]), a widely-used standard for anonymous data release. DP guarantees statistical difficulty in distinguishing whether or not a particular unit's data served as an input to a given algorithm, based on the algorithm's output. This protected unit may represent a single training example or a semantically higher-level unit like the entirety of a user's data.
While there are many methods for solving (1), we will follow Denisov et al. [10], Choquette-Choo et al. [7] and restrict to first-order algorithms  $\mathfrak{A}$  that linearly combine (stochastic) gradients. Each algorithm  $\mathcal{A} \in \mathfrak{A}$  is parameterized by a learning rate  $\gamma > 0$ , a number of steps  $T > 0$ , and scalars  $\{a_{tj}\}_{1 \leq j \leq t \leq T}$ . Given a starting point  $\mathbf{x}_0$ ,  $\mathcal{A}$  produces iterates  $\mathbf{x}_t \in \mathbb{R}^d$  given by
$$
\mathbf {x} _ {t + 1} = \mathbf {x} _ {0} - \gamma \mathcal {A} _ {t} (\mathbf {g} _ {1}, \ldots , \mathbf {g} _ {t}) \qquad \qquad \mathcal {A} _ {t} (\mathbf {g} _ {1}, \ldots , \mathbf {g} _ {t}) = \sum_ {j = 1} ^ {t} a _ {t j} \mathbf {g} _ {j}
$$
where  $\mathbf{g}_t$  is a (mini-batch) gradient of  $f$  computed at  $\mathbf{x}_t$ . This class encompasses a variety of first-order algorithms, including SGD [40], SGD with momentum [39, 35], and delayed SGD [2]. This class also captures algorithms that use learning rate scheduling, so long as the schedule is independent of the gradient values. We re-write the output of  $\mathcal{A}$  in matrix notation by defining:
$$
\mathbf {X} = \left[ \mathbf {x} _ {1}, \dots , \mathbf {x} _ {T} \right] ^ {\top} \in \mathbb {R} ^ {T \times d}, \mathbf {X} _ {0} = \left[ \mathbf {x} _ {0}, \dots , \mathbf {x} _ {0} \right] ^ {\top} \in \mathbb {R} ^ {T \times d}
$$
$$
\mathbf {G} = \left[ \mathbf {g} _ {1}, \dots , \mathbf {g} _ {T} \right] ^ {\top} \in \mathbb {R} ^ {T \times d}, \mathbf {A} = \left[ a _ {i j} \right] _ {1 \leq i, j \leq T} \in \mathbb {R} ^ {T \times T}
$$
Here  $\mathbf{A}$  is the workload matrix representing  $\mathcal{A}$ . At iteration  $t$ ,  $\mathcal{A}$  can only use the current and previous gradients, so  $a_{tj} = 0$  for  $j > t$  (ie.  $\mathbf{A}$  is lower-triangular). In this notation, the iterates of  $\mathcal{A}$  satisfy
$$
\mathbf {X} = \mathbf {X} _ {0} - \gamma \mathbf {A G}. \tag {2}
$$
Example 2.1 (SGD). Define the prefix-sum matrix  $\mathbf{S} \in \mathbb{R}^{T \times T}$  as the all-ones lower-triangular matrix. If  $\mathbf{A} = \mathbf{S}$ , then (2) is simply SGD with learning rate  $\gamma$ . As discussed by Denisov et al. [10, Section 4], we also recover SGD with momentum using an appropriate transformation  $\mathbf{S}'$  of  $\mathbf{S}$ .
2.1 Matrix Factorization and Privacy Mechanisms
In order to make the output of (2) differentially private, we typically need to clip the gradients and add noise. Let  $\overline{\mathbf{G}}$  denote the matrix whose rows (gradients) have been clipped to some  $\ell_2$  threshold  $\alpha$ . Let  $\mathbf{Z} \in \mathbb{R}^{T \times d}$  be a matrix with entries drawn independently from  $\mathcal{N}(0, \zeta^2 / d)$ . The well-known DP-SGD algorithm [1] adds this noise to each clipped gradient, so that
$$
\mathbf {X} = \mathbf {X} _ {0} - \gamma \mathbf {A} (\overline {{\mathbf {G}}} + \mathbf {Z}). \tag {3}
$$
For consistency, we consider (2) to be the special case of (3) where  $\mathbf{Z} = \mathbf{0}$  and  $\alpha = \infty$ . The variance  $\zeta^2$  depends on the clipping threshold  $\alpha$  and desired  $(\varepsilon, \delta)$  privacy we aim to achieve [1].
To derive algorithms with improved DP guarantees, Denisov et al. [10] add the noise  $\mathbf{Z}$  to a factorized version of  $\bar{\mathbf{A}}$ . For a factorization  $\mathbf{A} = \bar{\mathbf{B}}\mathbf{C}$  with  $\mathbf{B}, \mathbf{C} \in \mathbb{R}^{T \times T}$ , we add noise to the iterates via:
$$
\mathbf {X} = \mathbf {X} _ {0} - \gamma \mathbf {B} (\mathbf {C} \overline {{\mathbf {G}}} + \operatorname {s e n s} (\mathbf {C}) \mathbf {Z}) \equiv \mathbf {X} _ {0} - \gamma (\mathbf {A} \overline {{\mathbf {G}}} + \operatorname {s e n s} (\mathbf {C}) \mathbf {B} \mathbf {Z}). \tag {4}
$$
Here,  $\mathrm{sens}(\mathbf{C})$  is a number representing the sensitivity of the mapping  $\overline{\mathbf{G}}\mapsto \mathbf{C}\overline{\mathbf{G}}$  to "adjacent" input changes. We note that the sensitivity changes depending on the notion of adjacency. In single-epoch settings, two input matrices are adjacent if they differ by a single row [10], so the sensitivity function is  $\mathrm{sens}(\mathbf{C}):= \max_{i\in \{1,\dots,T\}}\| \mathbf{C}_{[i,i]}\| _2$ , i.e. the maximum  $\ell_2$ -squared column norm of  $\mathbf{C}$ . For details and extensions to multiple epochs, see [7].
If the variance of entries of  $\mathbf{Z}$  is fixed to some value  $\zeta^2 /d$ , then for all the possible factorizations  $\mathbf{A} = \mathbf{BC}$  in (4) have exactly same privacy guarantees, depending only on  $\zeta$ . It will also be convenient to define  $\sigma = \mathrm{sens}(\mathbf{C})\zeta$  as the 'effective' variance of  $\mathbf{Z}$  after re-scaling by the sensitivity. Note that for a fixed  $\sigma$ , the privacy guarantees of (4) might be different depending on the sensitivity.
The factorization  $\mathbf{B} = \mathbf{A}$ ,  $\mathbf{C} = \mathbf{I}$  recovers DP-SGD (3), but factorizations with better privacy-utility trade-offs may exist. The formulation of Eq. (4) transfers the linear optimization algorithm (2) into the setting of the matrix mechanism [26], a well-studied family of mechanisms in differential privacy. Denisov et al. [10], Choquette-Choo et al. [7] show that the mechanism in Eq. (4) provides a DP guarantee equivalent to a single application of the Gaussian mechanism, which can be computed tightly using numerical accounting techniques [49, 24].
Finding good factorizations. Intuitively, a factorization  $\mathbf{A} = \mathbf{BC}$  is good if  $\mathrm{sens}(\mathbf{C})$  is small and the added noise  $\mathbf{BZ}$  does not significantly degrade the convergence of (4). In order to quantify the effect of this added correlated noise on optimization, Denisov et al. [10] derive an online regret bound for (4) in the convex case against an adaptive adversary. Translating this via online-to-batch convergence to the stochastic setting, the iterates  $\mathbf{x}_t$  satisfy
$$
\frac {1}{T + 1} \sum_ {t = 0} ^ {T} \mathbb {E} [ f (\mathbf {x} _ {t}) - f ^ {\star} ] \leq \mathcal {O} \left(\frac {\| \mathbf {x} _ {0} - \mathbf {x} ^ {\star} \| ^ {2}}{\gamma T} + \gamma \tilde {L} ^ {2} + \gamma \zeta \tilde {L} \frac {\operatorname {s e n s} (\mathbf {C}) \| \mathbf {B} \| _ {F}}{\sqrt {T}}\right) \tag {5}
$$
where  $\tilde{L}$  is the Lipshitz constant of  $f$ . Denisov et al. [10] therefore use  $\mathrm{sen}(\mathbf{C})\| \mathbf{B}\| _F$  as a proxy for the impact of the factorized noise scheme on convergence. To find factorizations with good convergence properties, Denisov et al. [10], Choquette-Choo et al. [7] minimize  $\mathrm{sen}(\mathbf{C})\| \mathbf{B}\| _F$  subject to the constraint  $\mathbf{A} = \mathbf{BC}$ , which is equivalent to the following objective:
Problem 2.2 (Minimal-Norm Matrix Factorization). Given a lower triangular matrix  $\mathbf{A} \in \mathbb{R}^{T \times T}$ , define  $\mathrm{OPT}_F(\mathbf{A}) = (\mathbf{B}, \mathbf{C})$ , where  $\mathbf{B}, \mathbf{C} \in \mathbb{R}^{T \times T}$  solve the following optimization problem.
$$
\min  _ {\mathbf {B}, \mathbf {C}} \| \mathbf {B} \| _ {F} ^ {2} \text {s u c h t h a t} \mathbf {B C} = \mathbf {A}, \operatorname {s e n s} (\mathbf {C}) = 1. \tag {6}
$$
Eq. (6) is well-studied in the privacy literature and can be solved with a variety of numerical optimization algorithms [51, 30, 10, 7]. We also note that Denisov et al. [10] show that without loss of generality, we can assume  $\mathbf{B}$  and  $\mathbf{C}$  are lower triangular.
Finding improved factorizations. We argue that (5) is pessimistic in stochastic settings. For SGD (when  $\mathbf{B} = \mathbf{A}$ ), the last term in (5) is  $\mathcal{O}(\gamma \mathrm{sen}(\mathbf{C})\zeta \tilde{L}\sqrt{T})$ , which diverges with  $T$  for a constant stepsize. However, under the same assumptions as in [10], SGD with constant stepsize actually achieves a faster rate of  $\mathcal{O}(\gamma \mathrm{sen}(\mathbf{C})\zeta \tilde{L})$  (see [41]).
In this paper, we turn our attention to the smooth functions in order to focus on non-convex functions. We show in Appendix A, there are matrices  $\mathbf{B}_1, \mathbf{B}_2$  such that  $\mathrm{sens}(\mathbf{C}_1) \| \mathbf{B}_1 \|_F = \mathrm{sens}(\mathbf{C}_2) \| \mathbf{B}_2 \|_F$ , but Eq. (4) diverges with  $\mathbf{B}_1$  and converges with  $\mathbf{B}_2$ , therefore showing that Frobenius norm is not the right measure in the smooth case as well.
This begs the question of whether there are objectives that better capture the impact of the noise injected in (4) on convergence. To answer this, we derive a bound that can exhibit better dependence on  $\mathbf{B}$  to design better factorizations for differentially private optimization.
To study the effect of the noise BZ on optimization, we analyze a slightly simplified objective that omits parts of (4) not directly related to linear noise correlation. We do this as follows:
(I) We assume that each  $\mathbf{g}_t$  is the true gradient at the point  $\mathbf{x}_t$ , i.e.  $\mathbf{g}_t = \nabla f(\mathbf{x}_t)$ .
(II) We omit gradient clipping from our analysis. Alternatively, we can view this as setting the clipping threshold  $\alpha = \infty$  so that  $\overline{\mathbf{G}} = \mathbf{G}$  in (4).
(III) We restrict the class  $\mathcal{A}$  to SGD-type algorithms where  $\mathbf{A} = \mathbf{S}$ , as in Example 2.1.
We impose (I) for simplicity of presentation. Our results can be extended to stochastic gradients in a direct fashion. Restriction (II) is also for simplicity. First, clipping is not directly applied to the noise BZ. Second, for bounded domains or Lipschitz  $f$ , our analysis still holds with clipping. Last, practical DP methods often use adaptive clipping [45] instead of fixed clipping. We are not aware of convergence analyses for such schemes. We impose (III) in order to limit the class of algorithms A to a well-understood subclass. The convergence properties of (2) for general matrices A are not well-understood even when there is no noise  $(\mathbf{Z} = \mathbf{0})$ . As we discuss in Section 4, even with these simplifications, the effect of BZ is not well-understood.
Due to (III), we study factorizations  $\mathbf{BC}$  of the matrix  $\mathbf{A} = \mathbf{S}$ , as in Example 2.1. Then, (4) becomes
$$
\mathbf {X} = \mathbf {X} _ {0} - \gamma (\mathbf {S G} + \operatorname {s e n s} (\mathbf {C}) \mathbf {B Z}). \tag {7}
$$
In vector notation, for  $\mathbf{b}_0 = \mathbf{0}$  and  $\mathbf{B} = [\mathbf{b}_1,\dots \mathbf{b}_T]^\top$
$$
\mathbf {x} _ {t + 1} = \mathbf {x} _ {t} - \gamma [ \nabla f (\mathbf {x} _ {t}) + (\mathbf {b} _ {t + 1} - \mathbf {b} _ {t}) ^ {\top} \mathbf {Z} ], \tag {8}
$$
where for simplicity of presentation, we re-scaled the noise  $\mathbf{Z}$  by the sensitivity,  $\sigma^2 = \mathrm{sens}^2 (\mathbf{C})\zeta^2$  We now discuss several noteworthy special cases of (8).
Example 3.1 (PGD). If  $\mathbf{B} = \mathbf{S}$  (see Example 2.1) we recover SGD with uncorrelated additive noise, also known as perturbed gradient descent (PGD), where
$$
\mathbf {x} _ {t + 1} = \mathbf {x} _ {t} - \gamma [ \nabla f (\mathbf {x} _ {t}) + \mathbf {z} _ {t + 1} ]. \tag {9}
$$
The convergence rate of SGD (and therefore PGD) is well-understood in the optimization literature (e.g. see Bubeck [6, Section 6]).
Example 3.2 (Anti-PGD). By setting  $\mathbf{B} = \mathbf{I}$ , we get an algorithm that at every iteration adds an independent noise vector  $\mathbf{z}_{t+1}$  and subtracts the previously added noise  $\mathbf{z}_t$ :
$$
\mathbf {x} _ {t + 1} = \mathbf {x} _ {t} - \gamma [ \nabla f (\mathbf {x} _ {t}) + \mathbf {z} _ {t + 1} - \mathbf {z} _ {t} ], \quad \mathbf {z} _ {0} = \mathbf {0} \tag {10}
$$
Intuitively, this removes some of the noise added in the prior round. This is (up to a learning rate factor) the anti-correlated perturbed gradient descent (Anti-PGD) method proposed by Orvieto et al. [37], who study its generalization properties. Anti-PGD is also equivalent to SGD with randomized-smoothing [11]. The equivalence follows from defining  $\tilde{\mathbf{x}}_t = \mathbf{x}_t + \gamma \mathbf{z}_t$  and rewriting (10) as
$$
\tilde {\mathbf {x}} _ {t + 1} = \tilde {\mathbf {x}} _ {t} - \gamma \nabla f (\tilde {\mathbf {x}} _ {t} - \gamma \mathbf {z} _ {t}).
$$
While randomized smoothing algorithm is popular for non-smooth optimization, Vardhan and Stich [47] analyze its convergence properties in the smooth non-convex setting.
Example 3.3 (Tree Aggregation DP-FTRL). For  $k \geq 1$  and  $t = 2^{k - 1}$ , define  $\mathbf{H}_k \in \mathbb{R}^{(2^k - 1) \times t}$  recursively as follows:
$$
\mathbf {H} _ {1} = (1), \mathbf {H} _ {k + 1} = \left( \begin{array}{c c} \mathbf {H} _ {k} & \mathbf {0} \\ \mathbf {0} & \mathbf {H} _ {k} \\ \mathbf {1} & \mathbf {1} \end{array} \right)
$$
where  $\mathbf{1}$  above represents an all-ones row of appropriate width. For  $T = 2^{k - 1}$ , if  $\mathbf{C} = \mathbf{H}_k$  and  $\mathbf{B} = \mathbf{SC}_k^\dagger$  where  $\mathbf{C}_k^\dagger$  denotes a carefully chosen right pseudo-inverse of  $\mathbf{C}$ , then we recover the same noise matrix  $\mathbf{B}$  as in the DP-FTRL algorithm with either the online or full Honaker estimator (depending on the choice of  $\mathbf{C}^\dagger$ ) as in [20, 10]. Note that  $\mathbf{B}, \mathbf{C}$  are not square. This can be remedied by appropriately projecting onto  $\mathbb{R}^T$ . See Choquette-Choo et al. [7, Appendix D.3] for details.
4 Deriving Tighter Convergence Rates
We would like convergence rates for (7) that apply to any factorization and yield tight convergence rates for notable special cases. We pay special attention to PGD (Example 3.1) and Anti-PGD (Example 3.2), as they represent extremes in the space of factorizations ( $\mathbf{S} = \mathbf{SI}$  and  $\mathbf{S} = \mathbf{IS}$ , respectively). As we will show, it is possible to use existing theoretical tools to derive tight convergence rates for both, but not simultaneously.
Below, we discuss ways to derive tight rates for PGD and Anti-PGD, and how these rates involve incompatible analyses. We then develop a novel analytic framework involving restart iterates that allows us to analyze both methods simultaneously, as well as (7) for general factorizations. We start by formally stating our assumptions. For simplicity of presentation, we re-scale the noise  $\mathbf{Z}$  by the sensitivity of  $\mathbf{C}$ , i.e.  $\sigma^2 = \mathrm{sens}^2 (\mathbf{C})\zeta^2$ ; we will suppress the  $\mathbf{C}$  dependence of  $\sigma$ .
Assumption 4.1 (Noise). The rows  $\mathbf{z}_1, \ldots, \mathbf{z}_T$  of the noise matrix  $\mathbf{Z}$  are independent random vectors such that  $\forall t$ ,  $\mathbb{E}[\mathbf{z}_t] = \mathbf{0}$  and  $\mathbb{E}\left\|\mathbf{z}_t\right\|^2 \leq \sigma^2$ .
We do not assume  $\tilde{L}$ -Lipshitzness in our results, but we do assume  $L$ -smoothness. This is a relatively standard assumption in optimization literature [6].
Assumption 4.2 ( $L$ -smoothness). The function  $f: \mathbb{R}^d \to \mathbb{R}$  is differentiable, and there exists  $L > 0$  such that for all  $x, y \in \mathbb{R}^d$ ,  $\| \nabla f(\mathbf{x}) - \nabla f(\mathbf{y}) \| \leq L \| \mathbf{x} - \mathbf{y} \|$ .
For some of the results we will assume convexity.
Assumption 4.3 (Convexity). The function  $f: \mathbb{R}^d \to \mathbb{R}$  is convex, i.e.  $\forall \mathbf{x}, \mathbf{y} \in \mathbb{R}^d$ ,  $f(\mathbf{x}) - f(\mathbf{y}) \leq \langle \nabla f(\mathbf{x}), \mathbf{x} - \mathbf{y} \rangle$ . When assuming convexity, we also assume the infimum of  $f$  is achieved in  $\mathbb{R}^d$ .
4.1 Convergence Rates for PGD and Anti-PGD