Title: Near-Optimality of Contrastive Divergence Algorithms
Venue: NeurIPS 2024
Accepted: yes
Keywords: Statistical Estimation

Learning Theory

Stochastic Optimization

Exponential Families

Unnormalized Models
Abstract: We provide a non-asymptotic analysis of the contrastive divergence (CD) algorithm, a training method for unnormalized models. While prior work has established that (for exponential family distributions) the CD iterates asymptotically converge at an $O(n^{-1 / 3})$ rate to the true parameter of the data distribution, we show that CD can achieve the parametric rate $O(n^{-1 / 2})$. Our analysis provides results for various data batching schemes, including fully online and minibatch. We additionally show that CD is near-optimal, in the sense that its asymptotic variance is close to the Cramér-Rao lower bound.

Near-Optimality of Contrastive Divergence Algorithms

Abstract

Weperform a non-asymptotic analysis of the contrastive divergence (CD) algorithm, a training method for unnormalized models. While prior work has established that (for exponential family distributions) the CD iterates asymptotically converge at an O ( n -1 / 3 ) rate to the true parameter of the data distribution, we show, under some regularity assumptions, that CD can achieve the parametric rate O ( n -1 / 2 ) . Our analysis provides results for various data batching schemes, including the fully online and minibatch ones. We additionally show that CD can be near-optimal, in the sense that its asymptotic variance is close to the Cramér-Rao lower bound.

1 Introduction

Describing data using probability distributions is a central task in multiple scientific and industrial disciplines [1, 2, 3]. Since the true distribution of the data is generally unknown, such a task requires finding an estimator of the true distribution among a model class that best describes the available data. An estimator can be characterized at multiple levels of granularity: at the highest level lies consistency [4], a property which states that as the number of available data points increases, a given estimator will converge to the one best describing the data distribution. At a lower level, a consistent estimator can be further characterized by its convergence rate, a quantity upper-bounding its distance to the true distribution as a function of the number of samples. A convergence rate can be either asymptotic , e.g. hold only in the limit of an infinite sample size, or non-asymptotic , in which case the rate also holds for finite sample sizes. In their simplest form, convergence rates are provided in bigO notation, discarding finer grained information such as asymptotically dominated quantities as well as multiplicative constants. These constants play a role in the so-called asymptotic variance of the estimator, which is a precise descriptor of an estimator's statistical efficiency. Convergence rates and asymptotic variances have been the subject of extensive research in the statistical literature; in particular, well-known lower bounds exists regarding both the best possible (asymptotic) convergence rate of an estimator and its best possible asymptotic variance. These results set a clear frame of reference to interpret individual convergence rates, and are routinely present in the analysis of modern statistical algorithms such as noise-contrastive estimation [5, 6] or score matching [7, 8, 9].

In this work, we focus on cases where (1) the true data distribution admits a density with respect to some known base measure, and (2) the model class is parametrized by a finite-dimensional parameter. In this setting, provided that the true distribution belongs to the model class, a celebrated result in statistical estimation states that the model maximizing the average log-likelihood both achieves the best possible asymptotic convergence rate (called the parametric rate ) and the best possible asymptotic variance, called the Cramér-Rao bound (see, e.g. [10]). While this result shows that Maximum Likelihood Estimators (MLE) are asymptotically optimal, fitting them is complicated by computational hurdles when using models with intractable normalizing constants. Such unnormalized models are common in the Machine Learning literature due to their high flexibility [11, 12]; their weakness however lies in the fact that expectations under these models have no unbiased approximation. For this reason, popular approximation algorithms such as unbiased gradient-based

stochastic optimization of the empirical log-likelihood cannot a priori be used, as the gradient of the normalizing constant is given by an expectation under the model distribution.

The Contrastive Divergence (CD) algorithm [13] is a popular approach that circumvents this issue by using a Markov Chain Monte Carlo (MCMC) algorithm to approximate the gradient of the loglikelihood. Unnormalized models trained with Contrastive Divergence have been shown to reach competitive performance in high-dimensional tasks such as image [14, 15, 16], text [17], and protein modeling [18, 19], or neuroscience [20]. A consistency analysis of the Contrastive Divergence algorithm is delicate, however: indeed, the optimization error e.g. the difference between the estimate returned by CD and the MLE, is likely to be non-negligible as compared with statistical error - the distance between the MLE and the true distribution - and thus cannot be discarded, as often done when analyzing estimators that minimize tractable objectives [8, 5]. Recent work [21] elegantly established asymptotic O ( n -1 / 3 ) -consistency of the CD estimator for unnormalized exponential families when using only a finite number of MCMC steps. Key to their argument is the fact that the bias of the CD gradient estimate decreases as iterates approach the data distribution. However, as noted by the authors, their work left open the question of whether and under what conditions CD might achieve O ( n -1 / 2 ) -consistency.

Contributions In this work, we answer this question by providing a non-asymptotic analysis of the CD algorithm for unnormalized exponential families. While existing convergence bounds [21] were derived for the 'full batch' setting, where the CD gradient is estimated using the full dataset at each iteration, our analysis covers both the online setting (where data points are processed one at a time without replacement), and the offline setting with multiple data reuse strategies (including full batch).

In the online case (Section 3), we show, under a restricted set of assumptions compared to Jiang et al. [21], that the CD iterates can converge to the true distribution at the parametric O ( n -1 / 2 ) rate. Our analysis reveals that CD contains two sources of approximation: a bias term, and a variance term. These sources are almost independent of each other, in the sense that decreasing the bias by increasing the number of MCMC steps will not decrease the variance. The impact of these two sources of approximation transparently propagates in our resulting bounds: in particular, as the bias of the CD algorithm goes to 0, our bounds recover well-known results in online stochastic optimization [22]. Finally, we study the asymptotic variance of an estimator obtained by averaging the CD iterates, a classic acceleration technique in stochastic optimization [23]. We show that provided that the number of steps m is sufficiently large, the asymptotic variance of this estimator matches (up to a factor 4) the Cramér-Rao bound.

Next, we study the offline setting (Section 4), where the CD gradient is estimated by reusing (potentially random) subsets of a finite dataset. We show that a similar result to the online setup holds, up to an additional correlation term that arises from data reuse, and present several approaches to control this term. We improve over the results of [21] by showing a non-asymptotic and nearparametric rate at O ((log n ) 1 / 2 n -1 / 2 ) under their conditions, and also illustrate how different rates can be obtained under a variety of conditions. Our results also show an interesting tradeoff between the effect of initialization and the statistical error as a function of batch size.

In summary, we establish the near-optimality of a variety of Contrastive Divergence algorithms for unnormalized exponential families in the so called 'long-run' regime, where the number of MCMC steps is high enough to ensure that the CD gradient bias is sufficiently offset by the convexity of the negative log-likelihood.

2 Contrastive Divergence in Unnormalized Exponential Families

Unnormalized Exponential Families Exponential families (EF) [24, 25] form a well-studied class of probability distributions, given by

Here, X ∋ x is the data or sample space , which we set to be a subset of R d for some d ∈ N ∗ , although our results are readily extendable to more general measurable spaces. c is a measure on X called the base or carrier measure. When X ⊆ R d , c is often set to be the corresponding Lebesgue measure. ψ ∈ Ψ ⊆ R p is a finite-dimensional parameter called the natural parameter , and ϕ : R d ↦-→ R p is a function called the sufficient statistics , which, alongside with the base measure, fully describes an exponential family. Finally, log Z ( ψ ) , the log-normalizing (or cumulant ) function, is a quantity

ensuring that p ψ integrates to 1 over X . Crucially, we will not assume that log Z ( ψ ) admits a closed form expression for all ψ . The latter fact provides the practitioner with a great deal of flexibility in designing the model class: indeed, the only requirement that should be satisfied prior to performing statistical estimation is to have Z ( ψ ) < + ∞ for all ψ , something that can be readily verified and is often the case in practice. The drawback of unnormalized EFs is the fact that sampling (and thus approximating expectations under the model) cannot usually be performed in an unbiased manner. Instead, inference in unnormalized EFs is often performed using tools from the Bayesian Inference literature, such as MCMC [26]. Unnormalized EFs belong to the larger class of unnormalized models [27, 28, 7, 6], of the form e -E ψ ( x ) -log Z ( ψ ) c ( d x ) , Z ( ψ ) = ∫ e -E ψ ( x ) c ( d x ) , for some parametrized function E ψ : R d ↦-→ R referred to as the energy . Unnormalized models thus take the flexibility of unnormalized EFs one step further by allowing the (negative) unnormalized log-density to be an arbitrary function E ψ of x and ψ , instead of requiring a linear dependence on ψ as in Equation 1. We focus in this work on unnormalized EFs due to the multiple computational benefits they provide, as explained in the next section, but we believe that extending our analysis to more general unnormalized models is an interesting avenue for future work.

Statistical Estimation in Unnormalized Exponential Families using Contrastive Divergence We now review the Contrastive Divergence algorithm, an algorithm used to fit unnormalized models, and our main object of study in this work. The general setting is the following: we assume access to n i.i.d. samples ( X 1 , . . . , X n ) drawn from some unknown distribution p ⋆ , which we assume belongs to P ψ , e.g. p ⋆ = p ψ ⋆ for some ψ ⋆ ∈ Ψ . Given these samples, we aim to perform statistical estimation , e.g. find a parameter ψ n within Ψ that should approach ψ ⋆ as n grows.

The starting point of the Contrastive Divergence algorithm is the unfortunate realization that Maximum Likelihood Estimation, which corresponds to minimizing the cross-entropy L ( ψ ) := -E p n log d p ψ / d c between the model p ψ and the empirical data distribution p n := 1 /n ∑ n i =1 δ X i , cannot be performed using exact (possibly stochastic) gradient-based optimization, as the gradient ∇ ψ L ( ψ ) of L with respect to the parameter ψ contains an expectation under the model distribution p ψ . Indeed, the cross entropy and its gradient are given by

The second line follows from the well known identity ∇ ψ log Z ( ψ ) := E p ψ ϕ ; we refer to [25, Proposition 3.1] for a proof. The Contrastive Divergence algorithm circumvents this issue by running approximate stochastic gradient descent (SGD) on L , where the intractable expectation in ∇ ψ log Z is estimated using an MCMC algorithm initialized at the empirical data distribution. In more details, given a number of epochs T , a sequence of data batches B t,j of size B (e.g. B t,j ∈ [ [1 , n ] ] B , 1 ≤ t ≤ T, 1 ≤ j ≤ N ⌈ n/B ⌉ ), and a family of Markov kernels { k ψ , ψ ∈ Ψ } each with invariant distribution p ψ , at the j th minibatch of epoch t , ∇ ψ log Z ( ψ t, j -1 ) is approximated by 1 B ∑ i ∈ B t,j ϕ ( ˜ X i m ) , where ˜ X i m is produced by running the recursion ˜ X k i ∼ k ψ t ( ˜ X k -1 i , · ) , ˜ X 0 i = X i up to k = m . Throughout the paper, we will refer to the conditional distribution of ˜ X i m given X i as k m ψ ( X i , · ) . The resulting gradient estimate arising from combining this approximation with the other (tractable) sum over the data samples present in ∇ ψ L ( ψ ) , which we refer to as the CD gradient and denote as h t , is thus

Key to the behavior and analysis of the CD algorithm is the strategy employed to generate minibatches B t,j . The case where T = 1 , B = 1 , and B 1 ,j = { j } will be referred to as online CD, while the variant where T > 1 , and each batch B t,j draws B indices (with or without replacement) from [ [1 , n ] ] will be referred to as offline CD. In online CD, each data point is present in one and one batch only, while in offline CD, data points are reused across batches. From a statistical perspective, we will see that online CD can be analyzed in a remarkably simple way, while offline CD introduces additional correlations that require care to be controlled. Both settings come with their advantages and drawbacks, as we will see in the next section. The CD algorithms we study will employ decreasing step size schedules ( η t ) t ≥ 0 of the form η t = Ct -β , where C > 0 is the initial leaning rate and β ∈ [0 , 1] . We lay out online CD and offline CD in Algorithms 1 and 2. Note that our algorithms include a projection step on the parameter space Ψ to account for the case where Ψ is compact. In the case Ψ = R p , this step can be omitted. Next we depart from the setting of [21] and start by analyzing online CD.

Algorithm 1 Online CD

,

3 Non-asymptotic analysis of Online CD

3.1 Preliminaries and Assumptions

Recall that the chi-squared divergence between two probability measures p and q is defined as: χ 2 ( p, q ) := ∫ ( d p d q ( x ) -1) 2 q (d x ) if p ≪ q , and + ∞ otherwise. Here, p ≪ q denotes that p is absolutely continuous with respect to q and d p/ d q is the Radon-Nikodym derivative [29] of p with respect to q . Let L 2 ( p ψ ) be the space of square-integrable functions with respect to p ψ . For a function f ∈ L 2 ( p ψ ) , we define

which is a measure of how quick a Markov chain with kernel k ψ mixes, relative to the function f [30]. With these definitions in hand, we now state the assumptions required by our analysis of online CD. These assumptions form a strict subset of the assumptions considered in prior work [21], which required additional regularity and tail conditions on the Markov kernels k ψ .

Assumption A1. P ψ is a subset of a regular and minimal [25, Section 3.2] exponential family with natural parameter domain D ⊆ R p , Ψ is a convex and compact subset of D , and ψ ∗ lies in the interior of Ψ .

Assumption A2. There exists a constant C χ > 0 such that χ 2 ( p ψ ⋆ , p ψ ) ≤ C 2 ∥ ψ -ψ ⋆ ∥ 2

Assumption A3. α := sup { α ( f, ψ ) , f ∈ { ϕ i } i =1 ∪{ ϕ i ϕ j } i,j =1 , ψ ∈ Ψ } < 1 , where ϕ i is the i -th component of the function ϕ , and ϕ 2 is the i -th component of the function x ↦-→ ϕ ( x ) 2 .

χ p p i

A well known property of EFs [25, Proposition 3.1] is that their negative cross-entropy (against any other measure) is C ∞ , convex, and strictly so if the exponential family is minimal (meaning that the set of sufficient statistic functions ϕ i are not linearly dependent). Leaving aside the issue of intractable expectations, this convexity suggests that L can be efficiently minimized using stochastic approximation algorithms [31, 22]. The compactness of Ψ provided by Assumption A1 thus ensures, by the extreme value theorem [32], the existence of finite positive constants µ and L defined as:

where ∇ 2 ψ L is the Hessian of L with respect to ψ . µ (called the strong convexity constant) and L (a bound controlling the smoothness of the problem) play a critical role in the analysis of convex optimization algorithms [31]. While it is possible to obtain convergence rates in non-smooth or non-strongly-convex settings, our analysis follows the spirit of [21] by leveraging the strong convexity of the problem to compensate for the bias introduced by using CD gradients instead of unbiased stochastic gradients.

Assumption A2 allows link variations in distribution space to variations in parameter space, and will be instrumental to control the bias of the CD gradient. Note that since χ 2 ( p ψ ⋆ , p ψ ) =

Algorithm 2 Offline CD

e log Z (2 ψ -ψ ⋆ ) -(2 log Z ( ψ ) -log Z ( ψ ⋆ )) -1 provided that 2 ψ -ψ ⋆ ∈ D (see [33, Lemma 1]), we expect Assumption A2 to hold in many cases of interests. On the other hand, the possible exponential scaling of C χ w.r.t log Z suggests that this constant may be large in some instances.

Assumption A3 is a restricted spectral gap condition: it guarantees that the time required by the MCMCalgorithm to estimate expectations of ϕ and ϕ 2 under p ψ will be uniformly bounded. This assumption is weaker than the (unrestricted) uniform spectral gap condition of [21], which requires that α controls the convergence rate of all functions in L 2 ( p ψ ) . Note that standard results in stochastic analysis [34] guarantee that α ≤ 1 : thus, it only remains to ensure that α is strictly less than 1 . Spectral gaps are strongly dependent on two properties of distribution: their tail behavior and their multimodality. While multimodality poses the risk of pushing the constant α close to 1 , very heavy tails distributions may not verify the spectral gap condition at all.

3.2 Results

3.2.1 Parametric convergence of online CD

In this section, we show that under the assumptions stated in Section 3.1, the iterates ψ t produced by the online CD algorithm described in Algorithm 1 will converge to the true parameter ψ ⋆ at the parametric rate O ( n -1 / 2 ) . To do so, we follow a well known paradigm in convex optimization [22] by deriving a recursion on the quantity δ t := E ∥ ψ t -ψ ⋆ ∥ 2 , which will allow, after unrolling, to obtain convergence rates for the iterates ψ t . We aim to characterize precisely the impact of performing CD as opposed to performing online SGD on L , which would consist of replacing the CD gradient h t of Algorithm 1 by the unbiased (stochastic) gradient, given by:

which satisfies E g t ( ψ ) = ∇ ψ L ( ψ ) . The only stochasticity in g t comes from the sampling of a single data point x t from the true distribution, which is unavoidable in the online setting, and we have E ∥ g t ( ψ ⋆ ) ∥ 2 = Tr ( Cov p ψ⋆ ϕ ) =: σ 2 ⋆ . σ 2 ⋆ plays a key role in the analysis of Stochastic Gradient Descent [22]. We expect that replacing g t by h t will introduce two sources of approximation: a bias term coming from using a finite number of MCMC steps m , and an additional variance term, coming from using a single sample ˜ x t m to estimate ∇ ψ log Z ( ψ t ) . With that in mind, we derive a recursion on δ t in the following lemma.

Lemma 3.1. Let ( ψ t ) 0 ≤ t ≤ n be the iterates from Algorithm 1. Denote δ t = E ∥ ψ t -ψ ⋆ ∥ 2 , σ ⋆ = ( E p ψ⋆ ∥ ϕ -E p ψ⋆ ϕ ∥ 2 ) 1 / 2 , and σ t = ( E p ψt ∥ ϕ -E p ψt ϕ ∥ 2 ) 1 / 2 . Then, under A1, A2 and A3, for all t ≥ 1 ,

where ∥ log Z ∥ 3 , ∞ is a constant, ˜ µ m,t := µ -α m σ t C χ , and ˜ σ m,t := ( σ 2 ⋆ + σ 2 t +2 σ 2 t α 2 m ) 1 / 2 .

Lemma 3.1 is proved in Appendix D.2, which details the form of ∥ log Z ∥ 3 , ∞ , a constant that we expect to scale roughly as dL . Loosely speaking, this recursion suggests that as the learning rate η t goes to 0 , the two terms scaling in η 2 t will be negligible, in which case we will have: δ t ≤ (1 -2 η t ˜ µ m,t -1 ) δ t -1 < δ t -1 , yielding convergence of δ t to 0 . We make these arguments formal in the next theorem. The reader familiar with the convex optimization literature will note the similarities between this recursion and the one derived in [22], which would apply as is to online SGD on L using g t . The difference between the two recursions is that the roles of the strong convexity constant µ and the noise σ ⋆ are now played respectively by

These two modifications respectively characterize the impact of the bias and the additional variance introduced by the CD gradient. The last term in Equation 7, scaling in α m/ 2 η 2 t √ δ t , is a residual higher order mixed term coming from relating the variance of the Markov chain sample ˜ x t m to σ 2 t . This term can be easily controlled as done next, and disappears as m →∞ . Investigating the impact of m in the recursion, we notice that as m → ∞ , ˜ µ m,t → µ . As we will see later, this ensures that CD will converge for a sufficiently high m . On the other hand, in that same regime, ˜ σ m,t does not converge to σ ⋆ , but rather to ( σ 2 ⋆ + σ 2 t ) 1 / 2 , showing the irreducible impact of the variance term.

While we precisely investigate the impact of the residual variance term in the next section, we now unify σ ⋆ and σ t by introducing

̸

σ is an upper bound on the noise induced both by the CD gradient and by the online setup, and was used in prior work [21]. Note that by the properties of log Z , σ 2 also equals sup ψ ∈ Ψ tr( ∇ 2 ψ L ( ψ )) , where tr( A ) is the trace of A ∈ R p × p , and thus finite by the extreme value theorem. The following theorem is obtained by invoking standard unrolling arguments in the convex optimization literature. In the next result, we use the function φ γ ( t ) , defined as φ γ ( t ) = t γ -1 γ if γ = 0 , and log t if γ = 0 .

Theorem 3.2. Fix n ≥ 1 . Let ( ψ t ) 0 ≤ t ≤ n be the iterates produced by Algorithm 1, and define δ t := E ∥ ψ t -ψ ⋆ ∥ 2 . Moreover, assume that m > log( σC χ /µ ) log | α | , i.e. ˜ µ m := µ -α m σC χ > 0 . Then under Assumptions A1, A2 and A3, for η t = Ct -β with C > 0 , we have:

Theorem 3.2 is proved in Appendix D.3. It shows that the iterates produced by online CD will converge to the true parameter ψ ⋆ at the rate O ( n -1 / 2 ) provided that the number of steps m is sufficiently large, improving over the asymptotic O ( n -1 / 3 ) rate of [21], while imposing slightly weaker conditions on the number of steps m (see [21, Theorem 2.1]). This proves that online CD can be asymptotically competitive with other methods for training unnormalized models, such as Noise Contrastive Estimation [6], or Score Matching [7]. However, the asymptotic variance of ψ t (e.g. the multiplicative factor in front of the O ( n -1 / 2 ) term) is likely to be suboptimal, e.g. much larger than the Crámer-Rao bound, given by the trace of the inverse of the Fisher information matrix [25]. Given the statistical optimality of MLE, and the fact that CD in an approximate MLE method, this motivates the further goal or obtaining a CD estimator with near-optimal statistical properties. In the next section, we achieve this goal by showing that averaging the iterates ψ t will produce a near statistically-optimal estimator, in a sense that we will make precise.

3.2.2 Towards statistical optimality with averaging

Polyak-Ruppert averaging [23] is a simple yet surprisingly effective way to construct an asymptotically optimal estimator ¯ ψ n := 1 n ∑ n t =1 ψ i from a sequence of iterates ( ψ t ) 0 ≤ t ≤ n obtained by running a standard online SGD algorithm [22]. As shown in [22], when the objective is the cross-entropy of a model, and assuming the unbiased stochastic gradients are available, averaging yields an estimator ψ with the asymptotic variance tr( I ( ψ ⋆ ) -1 ) /n , where I ( ψ ) := Cov p ψ⋆ ϕ is the Fisher information matrix of the data distribution p ψ ⋆ . I ( ψ ⋆ ) -1 being the Cramér-Rao lower bound on asymptotic variances of statistical estimators [10], this estimator ¯ ψ n is asymptotically optimal. The following theorem shows conditions under which averaging CD iterates can give rise to a near-optimal estimator.

Theorem 3.3 (Contrastive Divergence with Polyak-Ruppert averaging) . Let ( ψ t ) t ≥ 0 the sequence of iterates obtained by running the CD algorithm with a learning rate η t = Ct -β for β ∈ ( 1 2 , 1) . Define ¯ ψ n := 1 n ∑ n i =1 ψ i . Then, under the same assumptions as Theorem 3.2, and assuming additionally that m := m ( n ) > (1 -β ) log n 2 | log α | , we have, for all n ≥ 1 ,

Consequently, we have that lim sup n →∞ n E ( ∥ ∥ ψ n -ψ ⋆ ∥ ∥ 2 ) ≤ 4tr( I ( ψ ⋆ ) -1 ) .

Theorem 3.3, alongside with a statement which includes the asymptotic order of the residual term, is proved in Appendix D.4. It shows that at the cost of an increase in computational complexity of the entire algorithm from O ( n ) to O ( n log n ) , ¯ ψ n will be a near-optimal statistical estimator of ψ ⋆ .

While this increase in complexity emerges from the bias of CD, the additional variance of CD results in an asymptotic variance inflated by a factor of 4 compared to the Cramér-Rao bound.

Theorem 3.3 concludes our analysis of online CD. Despite their asymptotic near-optimality, the bounds provided for online CD and its averaged version have weaknesses: the online CD iterates are not robust to choices of C. On the other hand, as shown in Appendix D.4, the bound of the averaged iterates contain higher-order terms that could be large in intermediate sample regimes. Next, we show that offline CD, which processes data points multiple times, can alleviate these issues.

4 Non-asymptotic analysis of offline CD

In practice, CD gradient approximation schemes are commonly used within an offline stochastic gradient descent (SGD) algorithm, where one is given the full sizen dataset upfront and each update uses some stochastic subset of the data. We study CD under offline SGD with replacement (SGDw), i.e. Algorithm 2 with batches B t,j being i.i.d. uniform draws of sizeB subsets of [ n ] , and include SGD without replacement in Appendix B.2. To do so, we follow the setting of prior work on offline CD [21], which established its asymptotic O ( n -1 3 ) consistency. We show that by slightly strengthening a moment assumption used in [21], the offline CD iterates converge to the true parameter at a near-parametric O ((log n ) 1 2 n -1 2 ) rate. Our proof proceeds by controlling a 'tail probability' term specific to the offline setting which characterizes the strength of the correlations between the offline CD iterates and the training data. While, as we show, the assumptions of [21] provide a tail control sufficient to obtain a near-parametric rate, other strategies are possible to obtain convergence guarantees. In particular, we show that non-asymptotic convergence can be obtained by either (1) relaxing assumptions on the Markov kernel required by prior work, or (2) making a specific mixing assumption the Markov chain.

4.1 Background: Asymptotic consistency of offline CD in subexponential settings

Prior work [21] has established asymptotic O ( n -1 3 ) consistency of the (averaged) offline CD iterates in the full-batch case. We summarize their results and assumptions below.

Assumption A4. There exists ν ≥ 2 s.t. for all m ∈ ◆ , there is κ ν ; m < ∞ s.t.

Assumption A5. There exists some C m > 0 such that, for all ψ 1 , ψ 2 ∈ Ψ , sup x ∈X ∥ E [ ϕ ( K m ψ 1 ( x ))] -E [ ϕ ( K m ψ 2 ( x ))] ∥ ≤ C m ∥ ψ 1 -ψ 2 ∥ .

Assumption A6. There exist some σ m , ζ m > 0 such that, for any z ∈ ❘ p with ∥ z ∥ ≤ ζ m , E [ e z ⊤ ( ϕ ( K m ψ ∗ ( X 1 )) -E [ ϕ ( K m ψ ∗ ( X 1 ))]) ] ≤ e σ 2 m ∥ z ∥ 2 / 2 .

Theorem 4.1 (Theorem 2.1 of [21]) . Assume assumptions A1,A2, A3, A4 (for ν = 2 ), A5 and A6. Let ψ t, 1 be the t -th iterate of offline CD with full-batch gradient descent and constant stepsize η t = C , i.e the iterates produced by Algorithm 2 using B t, 1 = [ [1 , n ] ] . Then for any learning rate C and number of Markov kernel steps m satisfying µ -α m σC χ -C 2 ( L + α m σC χ ) 2 > 0 , we have, for some A m > 0 ,

This result shows convergence of the averaged full-batch CD iterates to the true parameter in the large n and T limit. As discussed, this result is asymptotic both in n and T : the probability of the error exceeding A m n -1 3 goes to 0 as n →∞ and T →∞ , but at an unknown rate. Moreover, the O ( n -1 3 ) does not match the optimal O ( n -1 2 ) rate.

4.2 Sharpening offline CD bounds in subexponential settings

4.2.1 Non-asymptotic ˜ O ( n -1 / 2 ) -consistency

As a first result, we show that under the assumptions of [21] (except for a slightly stronger ν > 2 moment assumption in A4), ψ SGDw T,N in fact achieves a near-parametric rate. The most general version of our result holds for any learning rate schedule of the form Ct -β , β ∈ [0 , 1] , and for offline SGD

with arbitrary batch sizes B , with data drawn either with or without replacement across batches. For simplicity, we first present our result assuming full batch ( B = n, N = 1 , ψ SGDw t,j = ψ SGDw t, 1 for t ≥ 1 ) SGD with constant step sizes η t = C , which is the setting of [21]. Analogue bounds holding for the other mentioned batching and step sizes schedules can be found in Appendix B.

Theorem 4.2. Assume the setup of Theorem 4.1, except that Assumption A4 holds for some ν > 2 , and that ˜ µ m = µ -α m σC χ > 4 CL 2 . Let δ SGDw t,j := E ∥ ψ SGDw t,j -ψ ∗ ∥ 2 . Then, we have:

where E T, 1 1 , E T, 1 2 are functions decreasing exponentially in T , and C ′ ( p, ν, m, Ψ) is a constant in n, T . Consequently,

The precise values of all the constants can be found in Theorem B.1 (for E T, 1 1 , E T, 1 2 ) and Lemma B.3 (for C ′ ( p, ν, m, Ψ , β ) ), including their expressions for N > 1 and β ∈ [0 , 1] . We comment on the main differences between our result and the one of [21]. First our bound holds for any epoch T and number of samples n . Second, fixing n but taking T →∞ , the final bound matches the parametric O ( √ n ) up to a √ log( n ) factor, a significant improvement over the O ( n -1 3 ) rate of [21]. Finally, we control an L 2 error, which is a stronger control than a high probability bound by Markov's inequality; we hypothesize this is the reason why a slightly stronger moment assumption is required for our setup, compared to the one used for the high probability bound in [21].

Inspecting Equation 9, we notice the presence of two transient terms, and a stationary term , reminiscent of the structure of upper bound of Theorem 3.2. The transient terms (i.e. the ones containing E T, 1 1 and E T, 1 2 ) vanish exponentially fast in the total number of CD updates T . However, unlike in online CD where the number of updates and the number of samples are tied (e.g. T = n ), these two values are now decoupled , and these terms can be made arbitrarily small by increasing the number of gradient steps T without having to collect more samples n . The stationary term, which is the only one remaining in the limit of T →∞ , decreases with n at a rate that is independent of hyperparameters like the step size C or the learning rate schedule β (see Lemma B.3). In that sense, offline CD compares favorably to online CD, whose rate is sensitive to β and C , and averaged online CD, whose bound contains higher-order (in n ) terms which can be large in the moderate n regime. On the other hand, the stationary term in offline CD is asymptotically suboptimal: its rate is larger (while only up to a log factor) than the best-case O ( √ n ) one achieved by online CD algorithms, and the leading constant does not match the optimal one.

4.2.2 Proof of Theorem 4.2

The high-level proof of Theorem 4.2 follows a similar strategy as the online one: first, derive a recursion for the quantity δ SGDw t, 1 := E ∥ ψ SGDw t, 1 -ψ ∗ ∥ 2 , then unroll it explicitly to obtain a final bound on δ SGDw T, 1 . The main difference to online CD is the presence of an additional offline-specific correlation between the iterates and the data. We thus break down the proof into three steps: (1) deriving a controllable, uniform-in-time upper bound of the data-iterate correlations, (2) deriving and unrolling a recursion on δ SGDw t, 1 containing this new term, and (3) controlling that term to obtain a final bound on δ SGDw T, 1 .

̸

Step 1:characterizing the data-iterate correlations in offline CD In offline CD, at each epoch t ≥ 1 , the iterate ψ SGDw t -1 , 1 and the data samples X i are correlated: this is because these samples may have been used in previous epochs t ′ < t -1 to obtain the ψ SGDw t ′ , 1 , which themselves influenced ψ t -1 , 1 . With such correlations, we now have P ( X i | ψ SGDw t -1 , 1 ) = P ( X i ) , preventing us from obtaining an unrollable recursion on δ SGDw t, 1 by first marginalizing X i out to obtain an upper bound of E [∥ ∥ ψ SGDw t, 1 -ψ ∗ ∥ ∥ 2 | ψ SGDw t -1 , 1 ] that only depends on ∥ ∥ ψ SGDw t -1 , 1 -ψ ⋆ ∥ ∥ , and then marginalizing over ψ SGDw t -1 , 1 to obtain a recursion as in Lemma 3.1. As this problem would not have occurred had we used 'fresh samples' (e.g. i.i.d copies of X i not present in the training data) to perform our update, the

core of the proof lies in controlling the following quantity:

where X ′ 1 is an i.i.d. copy of X 1 . ∆( ψ SGDw t, 1 ) is the expected (over the data and iterates) error between a quantity that allows to obtain a recursion (the rightmost term) and the one actually used by offline CD (the leftmost term). To control it, we upper-bound it using a tail decomposition:

We invoke an additional assumption to ensure that ( E [∆( ψ SGDw t, 1 ) ν ]) 2 /ν is finite; in the results of [21], this is automatically implied by assumptions A4 and A6. For simplicity we assume the same bounding constant κ ν ; m .

Assumption A7. There exists ν ≥ 2 s.t. for all m ∈ ◆ , κ from A4 moreover verifies

Note the similarity of this assumption with assumption A4: the only difference is that X 1 is now a random training point instead of an deterministic (arbitrary) one. Ensuring assumption A7 in addition to assumption A4 thus requires controlling a ν -th order moment, instead of all moments as implied by assumption A6.

Step 2: Deriving and unrolling the recursion on δ SGDw t, 1 The right-hand side of Equation (10) does not depend on t , allowing for the derivation of an 'unrollable' recursion on δ SGDw t, 1 and its subsequent unrolling, which is performed in the following theorem. For simplicity, we again assume β = 0 and N = 1 and defer the general case to Theorem B.1 in appendix.

Theorem 4.3 (Convergence up to a tail control) . Assume A1, A2, A3, A4 and A7. Let η t = C for some C > 0 , and assume that ˜ µ m = µ -α m σC χ > 4 CL 2 . Then for any ϵ > 0 ,

where ε SGDw n,m,T ; ν ( ϵ ) is defined in Equation (10) .

Note that in the general, non-full batch B ≤ n case, 5 σ +5 κ ν,m √ n is replaced by 5 σ +5 κ ν,m √ B (see Theorem B.1). Under our bounds, obtaining consistency thus requires setting B ≡ B ( n ) → n →∞ + ∞ .

Step 3: Controlling the tail probability term Theorem 4.3 is just one step away from the final bound of Theorem 4.2: it remains to control the tail term ε SGDw n,m,T ; ν ( ϵ ) . Under the assumptions of [21], minimizing ε SGDw n,m,T ; ν ( ϵ ) over ϵ yields the following result:

Lemma 4.4. Assume the setup of Theorem 4.2. Let n ∈ ◆ be sufficiently large s.t. log n n < σ 2 m ζ 2 m p + ν -2 . Denote r Ψ as the radius of the smallest sphere in ❘ p that contains Ψ , which is finite under A1. Then inf ϵ> 0 ε SGDw ν ; n,m,T ( ϵ ) ≤

To obtain this result, we control the moment term (sup t E [∆( ψ SGDw t, 1 ) ν ]) using A7, and we control the tail probability term sup t /a80 (∆( ψ SGDw t, 1 ) > ϵ ) as in [21, Lemma 3.1] using an union bound, a covering argument and A6. Theorem 4.2 then follows by plugging Lemma 4.4 into Theorem 4.3.

4.3 Consistency of offline CD: beyond subexponential tails.

As discussed above, the general unrolling result of Theorem 4.3 holds without the subexponentiality assumption A6; this assumption was only used in Lemma 4.4 to control ε SGDw n,m,T ; ν ( ϵ ) . We now discuss two alternative ways to control this quantity without requiring subexponential tails. The first generalizes the idea of Jiang et al. [21], while the second exploits mixing of the Markov chain K m ψ ( x ) as m → ∞ . As before we only state partial results (full batch, β = 0 ) and defer the full explicit bounds to Appendix B.3.

Control via Markov Inequality Our first alternative uses Markov Inequality to yield the following. Theorem 4.5. Assume the setup of Theorem 4.3 and additionally that A5 holds. Then

where ˜ C and ˜ C ′ are functions whose explicit expressions are given in Lemma B.4 in the appendix.

In the case p = 1 and ν = 3 , the sub-optimal error from Theorem 4.5 reads O ( n -3 / 20 ) . Theorems 4.2 and 4.5 reveal that, depending on the tail condition imposed on the noise introduced by the Markov kernel, the convergence rate of offline CD varies: A subexponential tail, as assumed in prior work, in fact leads to near-parametric rate. Meanwhile, consistency can be obtained without assuming subexponentiality, albeit at a sub-optimal rate.

Control via Markov chain mixing. Alternatively, notice that E [∆( ψ SGDw t, 1 ) 2 ] involves an average of E [ ϕ ( K m ψ SGDw t, 1 ( X i ) 2 )∣ ∣ X i , ψ SGDw t, 1 ] -E [ ϕ ( K m ψ SGDw t, 1 ( X ′ 1 ) )∣ ∣ ψ SGDw t, 1 ] . When m →∞ , the effect of initialization vanishes, and one may expect the difference to converge to zero. We defer to Lemma B.5 in the appendix to show that, under a ϕ -discrepancy mixing condition ([35]) with a mixing coefficient ˜ α ∈ [0 , 1) ,

As m → ∞ , this recovers the parametric rate O ( n -1 / 2 ) . This alternative convergence guarantee comes at the cost of requiring m , the number of Markov chain steps, to grow with the sample size n .

Remark (Examples) . In our main results (Theorems 3.2, 3.3 and 4.3) and the tail condition for offline SGD (Theorem 4.2), we employed a weaker set of assumptions than those in [21] (except for the mild ν > 2 moment assumption in (A4)). Consequently, our results apply to all three examples studied in [21]: A bivariate Gaussian model with unknown mean and random-scan Gibbs sampler, a fully visible Boltzmann machine with random-scan Gibbs sampler, and an exponential-family random graph model with a Metropolis-Hastings sampler.

5 Related Work

Central to this paper is the prior work of Jiang et al. [21], which provided a rigorous theoretical foundation to analyze the convergence of full-batch CD, and which we refine. The study of optimization with biased gradient descent has attracted a lot of attention in recent years [36, 37, 38, 39]. These works, while closely connected to ours, analyze algorithms with different implementation choices than the CD algorithm: i.i.d. noise setup [36], or setup where a persistent Markov chain is maintained through the iterations [36, 37, 38, 39]. The latter is akin to a variant of the CD algorithm, called the persistent CD [40]. In contrast, our analysis focus on the CD algorithm that restarts a batch of Markov chains from the data distribution at every iteration. Finally, there is a rich body of work on convergence guarantees for offline multi-pass SGD [41, 42, 43, 44, 45, 46]. A notable difference of our analysis is that we are primarily concerned with statistical errors associated with convergence to the true parameter ψ ∗ in number of samples n , and not the commonly studied convergence rate in number of epochs T . Consequently, most of our work for the offline setup goes into handling the correlations that accumulate by reusing data across epochs.

6 Discussion

In this work, we provide a non-asymptotic analysis of the Contrastive Divergence algorithms, showing, in the online setting, their potential to converge at the parametric rate and to have near-optimal asymptotic variance, and proving a near-parametric rates in the offline setting, significantly extending prior results. Our results apply to unnormalized exponential families: despite their flexibility, these models only cover log-densities with linear relationships on the model parameters. We believe that extending our results to more general forms of unnormalized models is an important direction for future work.

Acknowledgments and Disclosure of Funding

All authors acknowledge support from the Gatsby Charitable Foundation.

References

Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences , 117(48):30055-30062, 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Aad W Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 2000.

Holden Lee, Chirag Pabbaraju, Anish Prasad Sevekari, and Andrej Risteski. Pitfalls of gaussians as a noise distribution in nce. In The Eleventh International Conference on Learning Representations , 2023.

Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , 2010.

Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 2005.

Frederic Koehler, Alexander Heckett, and Andrej Risteski. Statistical efficiency of score matching: The view from isoperimetry. In The Eleventh International Conference on Learning Representations , 2022.

Chirag Pabbaraju, Dhruv Rohatgi, Anish Prasad Sevekari, Holden Lee, Ankur Moitra, and Andrej Risteski. Provable benefits of score matching. Advances in Neural Information Processing Systems , 36, 2024.

George Casella and Roger L Berger. Statistical inference . Cengage Learning, 2021.

Nicolas Le Roux and Yoshua Bengio. Representational power of restricted boltzmann machines and deep belief networks. Neural computation , 20(6):1631-1649, 2008.

Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems , 32, 2019.

Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation , 2002.

Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of mcmc-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference on Artificial Intelligence , 2020.

Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcmc toward energy-based model. Advances in Neural Information Processing Systems , 2019.

Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy- based models. arXiv preprint arXiv:2021.01316 , 2021.

Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems , 35:9538-9551, 2022.

Deqian Kong, Bo Pang, Tian Han, and Ying Nian Wu. Molecule design by latent space energybased modeling and gradual distribution shifting. In Uncertainty in Artificial Intelligence , pages 1109-1120. PMLR, 2023.

Csilla Varnai, Nikolas S Burkoff, and David L Wild. Efficient parameter estimation of generalizable coarse-grained protein force fields using contrastive divergence: a maximum likelihood approach. Journal of chemical theory and computation , 9(12):5718-5733, 2013.

Pierre Glaser, Michael Arbel, Arnaud Doucet, and Arthur Gretton. Maximum likelihood learning of energy-based models for simulation-based inference. 2022.

Bai Jiang, Tung-Yu Wu, Yifan Jin, and Wing H Wong. Convergence of contrastive divergence algorithm in exponential family. The Annals of Statistics , 46(6A):3067-3098, 2018.

Eric Moulines and Francis Bach. Non-asymptotic analysis of stochastic approximation algorithms for machine learning. Advances in neural information processing systems , 24, 2011.

Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization , 30(4):838-855, 1992.

Lawrence D Brown. Fundamentals of statistical exponential families: with applications in statistical decision theory. Ims, 1986.

Martin J Wainwright, Michael I Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning , 1(1-2):1-305, 2008.

Charles J Geyer. Introduction to markov chain monte carlo. Handbook of markov chain monte carlo , 20116022:45, 2011.

Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energybased learning. Predicting structured data , 1(0), 2006.

Yang Song and Diederik P Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288 , 2021.

Joseph L Doob. Measure theory , volume 143. Springer Science & Business Media, 2012.

David A Levin and Yuval Peres. Markov chains and mixing times , volume 107. American Mathematical Soc., 2017.

Yurii Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2013.

Gary Harris and Clyde Martin. Shorter notes: The roots of a polynomial vary continuously as a function of the coefficients. Proceedings of the American Mathematical Society , pages 390-392, 1987.

Frank Nielsen and Richard Nock. On the chi square and higher-order chi distances for approximating f-divergences. IEEE Signal Processing Letters , 21(1):10-13, 2013.

Dominique Bakry, Ivan Gentil, Michel Ledoux, et al. Analysis and geometry of Markov diffusion operators , volume 103. Springer, 2014.

Maxim Rabinovich, Aaditya Ramdas, Michael I Jordan, and Martin J Wainwright. Functionspecific mixing times and concentration away from equilibrium. 2020.

Belhal Karimi, Blazej Miasojedow, Eric Moulines, and Hoi-To Wai. Non-asymptotic analysis of biased stochastic approximation scheme. In Conference on Learning Theory , pages 1944-1974. PMLR, 2019.

Tao Sun, Yuejiao Sun, and Wotao Yin. On markov chain gradient descent. Advances in neural information processing systems , 31, 2018.

Thinh T Doan. Finite-time analysis of markov gradient descent. IEEE Transactions on Automatic Control , 68(4):2140-2153, 2022.

Yves F Atchadé, Gersende Fort, and Eric Moulines. On perturbed proximal gradient algorithms. Journal of Machine Learning Research , 18(10):1-33, 2017.

Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning , 2008.

Andre Elisseeff, Theodoros Evgeniou, Massimiliano Pontil, and Leslie Pack Kaelbing. Stability of randomized learning algorithms. Journal of Machine Learning Research , 6(1), 2005.

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning , pages 1225-1234. PMLR, 2016.

Junhong Lin and Lorenzo Rosasco. Optimal rates for multi-pass stochastic gradient methods. Journal of Machine Learning Research , 18(97):1-47, 2017.

Loucas Pillaud-Vivien, Alessandro Rudi, and Francis Bach. Statistical optimality of stochastic gradient descent on hard learning problems through multiple passes. Advances in Neural Information Processing Systems , 31, 2018.

Nicole Mücke, Gergely Neu, and Lorenzo Rosasco. Beating sgd saturation with tail-averaging and minibatching. Advances in Neural Information Processing Systems , 32, 2019.

Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham Kakade. Risk bounds of multi-pass sgd for least squares in the interpolation regime. Advances in Neural Information Processing Systems , 35:12909-12920, 2022.

Dheeraj Nagaraj, Prateek Jain, and Praneeth Netrapalli. Sgd without replacement: Sharper rates for general smooth convex functions. In International Conference on Machine Learning , pages 4703-4711. PMLR, 2019.

Shashank Rajput, Anant Gupta, and Dimitris Papailiopoulos. Closing the convergence gap of sgd without replacement. In International Conference on Machine Learning , pages 7964-7973. PMLR, 2020.

Dimitri Bertsekas. Convex optimization theory , volume 1. Athena Scientific, 2009.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

S. W. Dharmadhikari, V. Fabian, and K. Jogdeo. Bounds on the moments of martingales. Ann. Math. Statist. , pages 1719-1723, 1968.

Supplementary Material for "Near-Optimality of Contrastive Divergence Algorithms"

The supplementary material provides the proofs of the main results of the paper:

Section B states full explicit bounds for the offline CD algorithm.

Section C collects a list of useful tools for our proofs. These include the properties of φ γ introduced before Theorem 3.2 in the main text, as well as several contraction and integrability results.

Section D provides the proofs for the online CD algorithm.

Sections E, F and G contain the proofs about the offline CD algorithm and the tail control.

A Notations

Throughout the proofs, we will denote by P m ψ the following operator from L 2 ( p ψ ) to itself:

Here, k m ( x, x ′ ) is the m -iterated version of some Markov transition kernel k ψ , e.g.:

Proj Ψ : R p ↦-→ Ψ denotes the projection operator onto the convex set Ψ , e.g.

We also frequently use the following function, used in standard convex optimization results [22].

̸

which is defined on R + \ { 0 } .

B Additional results for offline SGD

In this section, we provide the full statements on error bounds for SGD with replacement (SGDw), SGD with reshuffling (SGDo) and tail moment bounds, which complement the results in Section 4. Proofs are deferred to Appendix F, which make use of L 2 approximation by auxiliary gradient updates derived in Appendix E.

Notations Denote the SGDw iterates by ( ψ SGDw t,j ) t ∈ ◆ ,j ≤ N and let X ′ 1 be an i.i.d. copy of X 1 . Throughout the remaining of the appendix, we define given ϵ > 0 and n ∈ ◆

and ϑ SGDo n,m,T ( ϵ ) analogously. Using these notations, we can redefine the quantity ε SGDw n,m,T ; ν ( ϵ ) in the main as ε SGDw n,m,T ; ν ( ϵ ) := √ ϵ 2 + κ 2 ν ; m ( ϑ SGDw n,m,T ( ϵ ) ) ν -2 ν .

B.1 An explicit finite-sample bound for SGDw

In the result below, we write δ SGDw t,j := E ∥ ∥ ψ SGDw t,j -ψ ∗ ∥ ∥ 2 and, for a fixed ϵ > 0 , the quantity

Theorem B.1. Assume A1 (where Ψ may be non-compact), A2, A3, A4 and A7. Let η t = Ct -β for some β ∈ [0 , 1] and C > 0 , and assume that m > log( σC χ /µ ) log | α | s.t. ˜ µ m = µ -α m σC χ > 0 as in

Theorem 3.2. Then for any ϵ > 0 , √ δ SGDw T,N is upper bounded by

where E T,N 1 and E T,N 2 are two decreasing functions in T defined by

We emphasize that the full result above holds for any β ∈ [0 , 1] , which in particular includes the constant step size β = 0 regime considered by [21]. When β = 0 , for E T,N 1 and E T,N 2 to decay to zero as T →∞ , we additionally need the condition

This is almost identical to the condition used in [21, Equation 2.5, Theorem 2.1], except that 4 L 2 gets replaced by 1 2 ( L + α m σC χ ) 2 . Notably this says that an additional step size condition is needed for our results to hold in the constant step size regime, but not necessary for a decreasing step size.

B.2 Results for SGDo

SGD with reshuffling (SGDo, also called SGD without replacement) is an optimization scheme that is also widely used in practice compared to SGDo and online SGD. In the context of CD, it corresponds to Algorithm 2 with batches chosen as

where π is a uniform draw of the permutation group on n elements. We denote the iterates of SGDo ( ψ SGDo t,j ) t ∈ ◆ ,j ∈ [ N ] . Analogously to ϑ SGDw n,m,T , we define, for X ′ 1 an i.i.d. copy of X 1 , ϵ > 0 and n ∈ ◆ , the tail probability term

The following result says that ψ SGDo t,j enjoys exactly the same convergence guarantee as ψ SGDo t,j in Theorem 4.3. The statement is identical to that of Theorem B.1 and is stated in full for completeness; see Appendix F.2 for the proof, which is a slight adaptation of the proof for Theorem B.1. As before we write δ SGDo t,j := E ∥ ∥ ψ SGDo t,j -ψ ∗ ∥ ∥ 2 .

Theorem B.2 (Convergence of CD-SGDo) . Assume A1 (where Ψ may be non-compact), A2, A3, A4 and A7. Let η t = Ct -β for some β ∈ [0 , 1] and C > 0 , and assume that m > log( σC χ /µ ) log | α | s.t. ˜ µ m = µ -α m σC χ > 0 as in Theorem 3.2. Then for any ϵ > 0 , √ δ SGDo T,N is upper bounded by

where E T,N 1 and E T,N 2 are two decreasing functions in T defined by

Remark. We also remark that existing works [47, 48] show that the standard SGDo typically gives a faster convergence rate in T than SGDw. An analogous result for the CD setup would involve additional technical hurdles of jointly controlling the correlations across minibatches and from reusing data samples, and we defer this to future work.

B.3 Explicit tail control

We now provide the full explicit tail control bounds. All results in this section hold directly for ε SGDo ν ; n,m,T ( ϵ ) and δ SGDo T,N , and we omit them here. In the result below, we denote r Ψ as the radius of the smallest sphere in ❘ p that contains Ψ , which is finite under A1.

Lemma B.3. Assume A5 and A6. Let n ∈ ◆ be sufficiently large s.t. log n n < σ 2 m ζ 2 m p + ν -2 . Then inf ϵ> 0 ε SGDw ν ; n,m,T ( ϵ ) ≤

In particular, if we additionally assume the conditions of Theorem B.1, we have

where

Lemma B.4. Assume the conditions of Theorem 4.3 and additionally that A5 holds. Then

where

The next result considers a ϕ -discrepancy mixing condition ([35]), which is a mixing assumption on K m ψ but with respect to a specific test function ϕ , and we impose it uniformly over ψ ∈ Ψ . We also recall that X ψ 1 ∼ p ψ .

Lemma B.5. Assume that there exist ˜ α ∈ [0 , 1) and ˜ C K > 0 such that, for all ψ ∈ Ψ and x ∈ X , ∥ E [ ϕ ( K m ψ ( x ))] -E [ ϕ ( X ψ 1 )] ∥ ≤ ˜ C K ˜ α m . Then

In particular, if we additionally assume the conditions of Theorem 4.3, we have

C Auxiliary Tools

C.1 Properties of φ γ

The following lemma collects some identities used in [22].

Lemma C.1. φ γ satisfies the following properties:

(i) φ γ is increasing on R + for all γ ;

The next lemma provides some additional results on φ γ .

Lemma C.2. φ γ satisfies the following properties:

(i) φ γ is positive on t > 0 and increasing for every γ ∈ ❘ ;

(ii) For 1 ≤ t 1 ≤ t 2 and β ≥ 0 , we have

If instead β < 0 , we have

̸

(iii) For 1 ≤ t 1 ≤ t 2 and γ = 0 , we have

̸

(iv) Let 1 ≤ t 1 < t 2 and κ, β ≥ 0 . If κ = 1 and a > 0 , we have

̸

and if κ = 1 and a < 0 , we have

Proof of Lemma C.2. (i) follows from checking γ > 0 , γ = 0 and γ < 0 respectively. The first set of bounds in (ii) follow by noting that t ↦→ t -β is decreasing for β ≥ 0 :

The second set of bounds follows from noting that t ↦→ t -β is increasing for β < 0 :

̸

For (iii), we note that for γ = 0 , so by the mean value theorem,

The desired bounds then follow from an explicit computation of the infimum and the maximum in each of the two cases γ ≥ 1 and γ ≤ 1 .

̸

For (iv), we first consider the case κ = 1 and a > 0 . Then

Since, for x ≥ 0 , x ↦→ ( x +1) -κ is decreasing and x ↦→ exp( ax 1 -κ / (1 -κ )) is increasing, we have that for t 1 ≤ t ≤ t 2 and x ∈ [ t, t +1] ,

This implies that

̸

The main difference in the case κ = 1 and a < 0 is that we now use x ↦→ exp( a ( x +1) 1 -κ / (1 -κ )) is decreasing to obtain, for t 1 ≤ t ≤ t 2 and x ∈ [ t, t +1] ,

A similar argument then yields

We also need the following lemma, which is useful for controlling the accumulation of errors from the noise terms over iterations.

LemmaC.3. For any a, b ≥ 0 , T, N ∈ ◆ and κ, β ≥ 0 such that bt -β -at -κ ≤ 1 for all 1 ≤ t ≤ T , we have that

Moreover, for any ζ ≥ 0 , we have that

where and

In the special case ζ = β = 1 < κ , we have

Proof of Lemma C.3. By assumption, bt -β -at -κ ≤ 1 for all 1 ≤ t ≤ T . Since 0 ≤ 1 -x ≤ e -x for all x ≤ 1 , we have that for any 1 ≤ t 1 ≤ t 2 ≤ T ,

Applying this to the first quantity of interest followed by noting that a, b ≥ 0 and using Lemma C.2(ii), we obtain the first bound that

For the second bound, we define

̸

̸

Then by noting that 1 -bt -β + at -κ ≥ 0 for all 1 ≤ t ≤ T again,we can bound the quantity of interest as

In the last line, we have used that 0 ≤ 1 -b 2 t -β ≤ 1 for t ≥ t 0 +1 and 1 + at -κ ≤ 1 + a . To control the three quantities, we first note that by (13) , we have

In ( a ) above, we have noted that b 2 ≤ as -( κ -β ) for s ≤ t 0 ; in ( b ) , we have used t 0 ≤ T and Lemma C.2(ii) with a, b ≥ 0 . In the special case β = 1 < κ , the above yields

We now control S 2 . By (13) again, we have

̸

In ( c ) above, we have applied Lemma C.2(ii); in ( d ) , we have noted that sup t ∈ ◆ ( t +2) β /t β = 3 β . If κ = 1 and a > 0 , we can apply Lemma C.2(iv) to get that

If κ = 1 or a = 0 , the bound from ( c ) above reads

where we have used Lemma C.2(ii) in the last line. Now consider the special case with ζ = 1 < κ , the bound from ( d ) becomes

We are left with controlling S 1 , which follows from a similar strategy as controlling S 2 :

̸

In ( a ) above, we used Lemma C.2(ii). For β = 1 and b = 0 , we can apply Lemma C.2(iv) with b 2 > 0 to obtain

̸

In ( b ) , we have used Lemma C.2(iii) with 1 -β ≤ 1 . Meanwhile, if β = 1 or b = 0 , we have

For the special case with ζ = β = 1 , the bound in (14) becomes

In ( c ) , we have used Lemma C.2(ii) for both the case 1 -bN 2 ≤ 0 and 1 -bN 2 ≥ 0 .

Combining the bounds for the general cases, we obtain the first desired inequality that

For the special case ζ = β = 1 < κ, γ , combining the earlier bounds gives

C.2 Contraction and integrability results

The next result is a standard result in convex analysis, needed to handle projections performed in Algorithms 1 and 2.

Lemma C.4. Let Ψ a be convex subset of R p . Let ψ ⋆ ∈ Ψ . Then, for all ψ ∈ R p , we have:

Proof. We have:

Since by [49, Proposition 1.1.9], we have:

for all ψ ′ ∈ Ψ , we can use this inequality at ψ ′ = ψ ⋆ ∈ Ψ to obtain:

and the result follows by taking the square root.

We now state two lemmas that guarantee an amount of integrability sufficient to our analysis.

Lemma C.5. Let p, q ∈ P ( X ) such that d p d q exists, and such that χ 2 ( p ; q ) < + ∞ . Assume that f ∈ L 2 ( q ) Then | E p f | < + ∞ .

Proof. By assumption, f ∈ L 2 ( q ) . Moreover, χ 2 ( p, q ) < + ∞ , and thus we have d p d q -1 ∈ L 2 ( q ) . Thus, the inner product is finite, and

Lemma C.6. For all ψ ∈ Ψ , for all m ≥ 1 , and for all k ≥ 1 , we have:

Proof. By analycity of the log partition function ψ ↦-→ log Z ( ψ ) , we have ∫ ∥ ϕ ∥ k d p ψ ( x ) < + ∞ for all ψ ∈ Ψ , and thus, the function x ↦-→ ∥ ϕ ∥ k ( x ) ∈ L 2 ( p ψ ) for all ψ . Consequently, P m ψ ∥ ϕ ∥ k ∈ L 2 ( p ψ ) . We can apply Lemma C.5 to P m ψ ∥ ϕ ∥ k to obtain E ψ ⋆ P m ∥ ϕ ∥ k < + ∞ for all k ≥ 1 and for all m ≥ 0 . As a by-product, we obtain P m ψ ∥ ϕ ∥ k ∈ L 2 ( p ψ ⋆ ) , and thus so ∥ P m ϕ ∥ k .

The following lemma is used multiple time in our analysis.

Lemma C.7. Assume A3. Let q be a positive integer. Let f := ( f 1 , . . . , f q ) such that f k ∈ { ϕ i } p i =1 ∪ { ϕ i ϕ j } p i,j =1 for k ∈ [ q ] . Then, for all ψ ∈ Ψ , we have

Proof. Let us note first that

Here, we used the fact that P m ψ admits p ψ as an invariant measure in ( a ) [34, Eq. (1.2.2)], the Cauchy-Schwarz inequality in ( b )

C.3 Miscellaneous

Lemma C.8. Let f : Ψ → ❘ p be a differentiable function in the interior of Ψ ⊆ ❘ p . For ψ ∈ Ψ , define σ min ( ψ ) := inf θ ∈ Ψ , ∥ θ ∥ =1 θ ⊤ ∇ f ( ψ ) θ and σ max ( ψ ) := sup θ ∈ Ψ , ∥ θ ∥ =1 θ ⊤ ∇ f ( ψ ) θ with respect to the Jacobian matrix ∇ f ( ψ ) . Then for any ψ 1 , ψ 2 ∈ Ψ , we have that

Proof of Lemma C.8. By the mean value theorem, there exists some a ∈ (0 , 1) such that

Plugging in the definition of σ max gives the desired upper bound and similarly σ min implies the lower bound.

D Proofs for Online CD

D.1 Auxiliary Lemmas for Online CD

We recall the following notations used in the next lemmas, namely σ ψ := E p ψ ∥ ϕ -E p ψ ϕ ∥ 2 , σ ⋆ := σ ψ ⋆ and σ := sup ψ ∈ Ψ σ ψ .

We now provide two intermediary lemmas necessary to analyze the impact of variance in the CD gradient. The strategy is similar in both of them: we change the integration from p ψ ⋆ to p ψ to obtain contraction, at the cost of an additional term scaling with C χ ∥ ψ -ψ ⋆ ∥ . We obtain exact constants that we choose to describe in terms of the smoothness parameters of the problem, e.g. the k th derivatives of the log partition function log Z , which, for k ≥ 2 , equals the k th derivative of the negative cross-entropy model w.r.t p ψ ⋆ .

Second Moment convergence The following lemmas guarantee the second moment of a sample from k m ψ p ψ ⋆ approaches the second moment of a sample from the target distribution p ψ .

Lemma D.1. Under A1, A2 and A3, for all ψ ∈ Ψ , we have:

where

Proof. Applying Lemma C.7 to each f i := ϕ 2 i , we have

We map the two moments to derivatives of log Z ( ψ ) , since the k th derivative of log Z ( ψ ) is the k th cumulant. It can be shown, using the multivariate moment to cumulant mapping, that

where ∂ k i log Z ( ψ ) denotes the k th derivative of log Z with respect to ψ i . On the other hand,

implying

The result follows by summing over i , since:

Note that ∥ log Z ∥ 1 , ∞ is finite since Ψ is compact and log Z is analytic.

Squared First Moment convergence The next lemma provides convergence (in squared absolute value) of the first moment of the m -iterated Markov kernel k m ψ .

Lemma D.2. Under A1, A2 and A3, for all ψ ∈ Ψ , we have

where and

Proof. We have:

where

In ( a ) , we used the restricted spectral gap Assumption A3 for ∆ 1 , 1 , and the Cauchy-Schwarz inequality combined with Assumption A2 for ∆ 1 , 2 . In ( b ) , we used the Cauchy-Schwarz once again, and in ( c ) we used the fact that P m ψ is a contraction in L 6 ( p ψ ) and another invocation of the spectral gap assumption A3 As an aside, note that a simpler result can be obtained by making regularity assumption on the mapping ψ ↦-→ P m ψ . Assuming that ψ ↦-→ P m ψ ( x ) is uniformly L m -Lipschitz across x ∈ X for instance (as done in [21, Assumption 5]), the second term ∆ 1 , 2 of ∆ 1 could have been handled using

Although this result does not require possibly large constants related to sixth-order moments, it is less tight in the sense that it does not go to 0 as m →∞ . Back to the main proof, and ∆ 2 in particular. Applying Lemma C.7 to f := ϕ , we have

Putting everything together, we have:

Summing over i , we obtain

where

Similarly to the previous lemma, one can upper bound E ( ϕ i -E p ψ ϕ i ) 6 using the centered moment to cumulant formula:

To get a full description of ∥ log Z ∥ 2 , ∞ :

Lemma D.3. Under A1, A2 and A3, for all ψ ∈ Ψ , we have

where and

Proof. We have:

where

In ( a ) , we used the restricted spectral gap Assumption A3 for ∆ 1 , 1 , and the Cauchy-Schwarz inequality combined with Assumption A2 for ∆ 1 , 2 . In ( b ) , we used the Cauchy-Schwarz once again, and in ( c ) we used the fact that P m ψ is a contraction in L 6 ( p ψ ) and another invocation of the spectral gap assumption A3 As an aside, note that a simpler result can be obtained by making regularity assumption on the mapping ψ ↦-→ P m ψ . Assuming that ψ ↦-→ P m ψ ( x ) is uniformly L m -Lipschitz across x ∈ X for instance (as done in [21, Assumption 5]), the second term ∆ 1 , 2 of ∆ 1 could have been handled using

Although this result does not require possibly large constants related to sixth-order moments, it is less tight in the sense that it does not go to 0 as m →∞ . Back to the main proof, and ∆ 2 in particular. Applying Lemma C.7 to f := ϕ , we have

Putting everything together, we have:

Summing over i , we obtain

where

Similarly to the previous lemma, one can upper bound E ( ϕ i -E p ψ ϕ i ) 6 using the centered moment to cumulant formula:

To get a full description of ∥ log Z ∥ 2 , ∞ :

We can now use the previous lemmas to obtain an expression on the second moment of the contrastive divergence gradient estimator, relating it to the one of the stochastic log-likelihood gradient estimator. Lemma D.4. Under A1, A2 and A3, we have:

3 , ∞ 1 , ∞ 2 , ∞

Proof. We rely on the following decomposition:

∆ 1 , 1 +∆ 1 , 2 form the differentiable stochastic gradient g t of Equation 6. Note that ∆ 1 , 1 is meanzero, and ∆ 2 is mean-zero conditionally on X t . Consequently, E ⟨ ∆ 2 , ∆ 3 ⟩ = E ⟨ ∆ 2 , ∆ 1 , 1 ⟩ = E ⟨ ∆ 1 , 1 , ∆ 1 , 2 ⟩ = E ⟨ ∆ 1 , 1 , ∆ 2 ⟩ = 0 , and the only mixed-terms that remain to be controlled are E ⟨ ∆ 1 , 1 , ∆ 3 ⟩ and E ⟨ ∆ 1 , 2 , ∆ 3 ⟩ . We first control the unmixed terms, and the simple ones first: we have E ∥ ∆ 1 , 1 ∥ 2 = σ 2 ⋆ , as well as E ∥ ∆ 1 , 2 ∥ 2 ≤ L 2 ∥ ψ -ψ ⋆ ∥ 2 . For ∆ 2 , we have:

We can invoke Lemmas D.3 and D.1, which guarantee

to obtain

where ∥ log Z ∥ 3 , ∞ := 2 max( ∥ log Z ∥ 1 , ∞ , ∥ log Z ∥ 2 , ∞ ) .

For ∆ 3 , notice that ∆ 3 is precisely the term ∆ 1 in Lemma D.3, and we can thus bound it by

Finally, we simply bound 2 E ⟨ ∆ 1 , 1 , ∆ 3 ⟩ by E ∥ ∆ 1 , 1 ∥ 2 + E ∥ ∆ 3 ∥ 2 , and 2 E ⟨ ∆ 1 , 2 , ∆ 3 ⟩ by E ∥ ∆ 1 , 2 ∥ 2 + E ∥ ∆ 3 ∥ 2 . Putting everything together, we have:

D.2 Proof of the SGD recursion (Lemma 3.1)

We are now ready to provide an SGD-style recursion for the expected squared distance to the optimum E [ ∥ ψ t -ψ ⋆ ∥ 2 ] .

Lemma (Restatement of Lemma 3.1) . Let ψ t be the iterates produced by Algorithm 1. Denote δ t = E [ ∥ ψ t -ψ ⋆ ∥ 2 ] , σ ⋆ = ( E p ψ⋆ ∥ ϕ -E p ψ⋆ ϕ ∥ 2 ) 1 / 2 , and σ t = ( E p ψt ∥ ϕ -E p ψt ϕ ∥ 2 ) 1 / 2 . Then, under Assumptions A1, A2 and A3, for all t ≥ 1 , we have:

Proof. In this proof, we note ( F t ) t ≥ 0 , the increasing family of σ -algebras generated by the random variables ( X t ) t ≥ 0 ∼ p ψ ⋆ and the Markov chain samples ˜ X t m | X t , ψ t ∼ k m ψ t ( X t , · ) . We decompose the integrand of δ t as follows:

The first term is (up to an averaging operation) the previous iterate. The middle term will ensure (provided m is large enough) contraction of the expected distance to the optimum. Finally, the third term can be described by Lemma D.4, and essentially behaves like the second moment of a log-likelihood stochastic gradient. Indeed, noting g ( ψ ) := -E p ψ⋆ ϕ + E p ψ ϕ the expectation of g t w.r.t x t (which is the gradient of the negative cross-entropy between p ψ and p ψ ⋆ ), we have:

and applying Lemma C.7, we get that

meaning

On the other hand, by applying Lemma C.8 to g , we have:

Combining the above results, we obtain:

And the result follows by integrating over F t -1 .

D.3 Proof of Online CD convergence

We now prove Theorem 3.2. The recursion of Lemma 3.1 is almost identifiable, up to a cross-term of second order, with the one of an SGD algorithm as presented in the setting of [22, Theorem 1]. To make the identification exact, we use the bound 4 α m/ 2 η 2 t ∥ log Z ∥ 3 , ∞ C χ δ 1 / 2 t -1 ≤ 2 α m/ 2 η 2 t δ t + 2 α m/ 2 η 2 t ∥ log Z ∥ 2 3 , ∞ C 2 χ , yielding the following recursion:

where we used the fact that ˜ σ m,t ≤ σ . This recursion is of the same form as the one studied in [22, Equation 6, Theorem 1] given by:

We can use the same unrolling strategy as theirs (the only condition required to proceed is that ˜ µ m > ˜ L , which automatically holds since µ < L ), and we obtain

by identifying:

D.4 Proof of online CD with averaging (Theorem 3.3)

We first restate the theorem in its complete form.

Theorem (Contrastive Divergence with Polyak-Ruppert averaging) . Let ( ψ t ) t ≥ 0 the sequence of iterates obtained by running the CD algorithm with a learning rate η t = Ct -β for β ∈ ( 1 2 , 1) . Define ¯ ψ n := 1 n ∑ n i =1 ψ i . Then, under the same assumptions as Theorem 3.2 we have, for all n ≥ 1 ,

Where I ( ψ ⋆ ) := Cov p ψ⋆ [ ϕ ] is the Fisher information matrix of the data distribution. Additionally, if .

Throughout the proof, we will denote by h n the standard online CD gradient defined in Equation 3:

as well as

We start by establishing some intermediate lemmas.

Lemma D.5. Under Assumptions A1, A2, A3, the online CD iterates produced by Algorithm 1 using η t = Ct -β for β ∈ ( 1 2 , 1) verify

Proof. Let us note δ n = E [ ∥ ψ n -ψ ⋆ ∥ 2 ] . Summing the r.h.s of Theorem 3.2, we have

where A 3 is finite if β < 1 , and A 3 = O ( n ) otherwise [22].

Lemma D.6. Under Assumptions A1, A2, A3, the online CD iterates produced by Algorithm 1 using η t = Ct -β for β ∈ ( 1 2 , 1) verify

Proof. Let us note δ n = E [ ∥ ψ n -ψ ⋆ ∥ 2 ] . Applying √ x + y ≤ √ x + √ y to the r.h.s of Theorem 3.2, we have

where A 4 is finite if β < 1 , and A 4 = O ( n ) otherwise [22].

Lemma D.7. Under Assumptions A1, A2, A3, the online CD iterates produced by Algorithm 1 using η t = Ct -β for β ∈ ( 1 2 , 1) verify

Proof. The result follows from the fact that h t verifies

A similar quantity was handled in the case of standard SGD [22, Theorem 3], and the only condition needed to reuse their steps is that ( ψ t ) t ≤ n satisfies an upper bound of the same form as the one [22, Theorem 1] derived. This is precisely the nature of our bound of ψ t estblished in Theorem 3.2, with ˜ µ m , ˜ σ m , ˜ L . Borrowing on their result, we have:

where ˜ µ m , ˜ σ m and ˜ L are defined in 3.2, and

Lemma D.8. Under Assumptions A1, A2, A3, the online CD iterates produced by Algorithm 1 using η t = Ct -β for β ∈ ( 1 2 , 1) verify

Proof. We proceed as in the proof of [22, Theorem 3], first establishing a recurrence for E [ ∥ ψ i -ψ ⋆ ∥ 4 ] , and then unrolling it. We have

The second and fourth terms will be controlled using results from our previous sections. For simplicity, we don't attempt to relate the moments of ∥ h n ∥ 4 as precisely as before. Instead we use:

And let us note τ = ( sup ψ ∈ Ψ E p ψ ∥ ϕ ∥ 8 ) 1 / 8 . We have, for k ≥ 4 ,

Where we used the fact that P m ψ n -1 is a contraction, and ( E ∥ ϕ ∥ k ) 1 /k is an increasing function of k . On the other hand, we simply have E p ψ⋆ ∥ ϕ ( X n , · ) ∥ k ≤ τ k . Plugging this into the previous equation, we obtain

To simplify the recursion, we use the four following inequalities:

Injecting them in our recursion, we obtain:

which, after further simplifications, yields

where we defined ˜ τ 1 := 2(1 + C χ ) τ and ˜ L 1 := 2(1 + C χ ) τ + L . This recursion is of the form of the one studied in [22, Equation 32] (note that by design, ˜ L ≥ ˜ µ m .) The steps performed to bound E [ ∥ ψ i -ψ ⋆ ∥ 4 ] thus follow from their derivations, and we obtain:

where

and we have A (1) < + ∞ if β < 1 , and A (1) = O ( n ) otherwise.

Lemma D.9. For all ψ ∈ Ψ , we have:

Proof. We have

Looking at the second moment first, we have

Applying lemma C.7 to the R d 2 -valued function f given by f ij := ϕ i ϕ j -E p ψ ϕ i ϕ j ,

We now investigate the first moment. We have

and thus, applying Lemma C.7 on ∆ 2 , 1 , we have:

where we used that ∥ ∆ 2 , 1 ∆ ⊤ 2 , 1 ∥ F = ∥ ∆ 2 , 1 ∥ 2 . We can now combine our two matrix moment bounds to obtain

Lemma D.10. Under Assumptions A1, A2, A3 it holds that

where M := sup ψ ∈ Ψ ∥ ∥ ∇ 3 log Z ( ψ ) ∥ ∥ op( ∥·∥ F , ∥·∥ F ) < + ∞ .

Noting ∆ the l.h.s of Lemma D.10, we have, summing over [ n ] , and using Minkowski's inequality,

Note that this step was made possible because we are looking at the square-root of the variance, which is unlike the recursion in Lemma 3.1. This allows to separate the terms and use fewer intermediaries than in the proof of Lemma 3.1.

Since both ∆ 1 ,n and ∆ 2 ,n are martingale differences with respect to the filtration F n -1 , the covariance terms vanish, and we have

In ( a ) we used Lemma D.10, the cyclicity of the trace, tr( A ⊤ B ) ≤ ∥ AB ∥ F ≤ ∥ A ∥ F ∥ B ∥ F , and the fact that since Cov p ψ i -1 [ ϕ ( x i )] = ∇ 2 ψ L ( ψ i -1 ) , by analycity of L , there exists a constant M := sup ψ ∈ Ψ ∥ ∥ ∇ 3 log Z ( ψ ) ∥ ∥ op( ∥·∥ , ∥·∥ F ) such that

In ( b ) we used Jensen's inequality to get E [ ∥ ψ i -1 -ψ ⋆ ∥ ] ≤ √ E [ ∥ ψ i -1 -ψ ⋆ ∥ 2 ] = δ 1 / 2 i -1 .

We are now ready to prove Theorem 3.3.

Proof of Theorem 3.3. It holds that:

Applying on both sides: (a) a summation over i ∈ [ n ] , (b) a multiplication by f ′′ ( ψ ⋆ ) -1 , (c) √ E [ ∥ · ∥ 2 ] , and using Minkowski's inequality on the r.h.s, we obtain

( i ) and ( ii ) have direct analogues in the proofs of prior work [22] on the convergence of (unbiased) SGD with Polyak-Ruppert averaging, and will be bounded similarly. ( iii ) captures the bias of the CD algorithm, while ( iv ) captures the variance.

Bounding ( i ) Using Lemma D.7, we have ( i ) = O ( n β 2 -1 ) .

Bounding ( ii ) Since log Z ( ψ ) is analytic, there exists some constant M ′ such that

Thus, we have:

where the second-to-last inequality used Minkowski's inequality, and the last applied Lemma D.8.

Bounding ( iii ) By Minkowski's inequality, we have:

Moreover, using lemma C.7, we have:

We thus obtain

Recalling that δ i satisfies Theorem 3.2, we have that δ 1 2 n = O ( n -β/ 2 ) , and we thus have ∑ n i =1 δ 1 / 2 i = O ( n 1 -β 2 ) . By squaring the result of Lemma D.6, we have ∑ n i =1 δ 1 / 2 i = O ( n -1 -β 2 ) , and thus, we obtain that ( iii ) = O ( α m n -β/ 2 ) = O ( n -( β 2 + m | log α | log n ) ) .

Bounding ( iv ) We have

Where in ( a ) , we used Lemma D.10 and in ( b ) , we used Lemma D.6 and D.5.

Final bound Putting everything together, we have that:

If, furthermore, m> (1 -β ) log n 2 | log α | , we have

which concludes the proof.

E L 2 approximation by auxiliary gradient updates

In this section, we consider different gradient update schemes starting from some random initialization θ init , and control the L 2 distance between the different updates and the deterministic target ψ ∗ ∈ Ψ .

Notation. Recall the notation that X 1 , . . . , X n i . i . d . ∼ p ψ ∗ , B = n/N and for ψ ∈ Ψ , K ψ ( x ) ∼ k m ψ ( x, · ) . We also write, for m ∈ ◆ ∪ {∞} ,

Let θ init be some Ψ -valued random initialization that is possibly correlated with X 1 , . . . , X n . We capture the effect of correlation through the following quantities: For ϵ > 0 and ν > 2 , let

We also consider the i.i.d. samples, drawn independently of X 1 , . . . , X n and on a given ψ ∈ Ψ , as

For notational clarity, we shall use θ m,B to denote parameters arising from an one-step update, where the subscripts m,B represent performing the one-step update with lengthm Markov chains and with batch size B . This is to be distinguished from ψ t elsewhere in the text, which denotes the parameter from the actual multi-step CD algorithm and the subscript t denotes the t -th CD iterate.

Gradient update schemes. We consider five different updates. Let X ′ 1 be an i.i.d. copy of X 1 drawn independently of all other random variables. The SGD-with-replacement update is given by

and S w is a uniformly drawn sizeB subset of [ n ] . The SGD-without-replacement update, after renormalizing the learning rate, is given by the N -fold function composition

and S o j 's are disjoint sizeB random subsets of [ n ] , defined by ( S o 1 , . . . , S o N ) = π ([ n ]) for a uniformly drawn element π of the permutation group on n objects. The full-batch gradient update is given by

The full-batch gradient update with an infinite-length Markov chain is given by

The population gradient update with an infinite-length Markov chain is given by

where we use the lowercase f to emphasize that f pop is a deterministic function.

The forthcoming results are summarized below:

Lemma E.1. Let F n be the sigma algebra generated by { X i , K m i ; θ init ( X i ) | 1 ≤ i ≤ n } . Then

Moreover, under A1 and A7, we have

Proof of Lemma E.1. Write A := ( A 1 , . . . , A n ) , where

Since S w is uniformly drawn from all sizeB subsets of [ n ] and independently of all other variables, we have that almost surely

To prove the remaining bound, we note that the above relation implies θ SGDw m,B -θ GD m is zero-mean. By the law of total variance, we have

To compute the covariance, recall that S w is a uniformly drawn sizeB subset of [ n ] with B = n/N . Let P N ([ n ]) be the collection of all partitions of [ n ] into N sizeB subsets. We can generate S w by the following two-step process:

(i) Uniformly draw a partition P ′ = ( P ′ 1 , . . . , P ′ N ) from P N ([ n ]) ;

(ii) Uniformly sample an index K from [ N ] and set S w = P ′ K .

Then we have, almost surely

By noting that A i 's are exchangeable, we obtain

In ( a ) , we have used a Cauchy-Schwarz inequality; in ( b ) , we have used A1 and A7. Finally we note that if B = n , θ SGDw m,B = θ GD m almost surely, which implies the desired bound.

Lemma E.2. Denote A n as the sigma algebra generated by θ init , X 1 , . . . , X n . Under A1, A2, A3 and A7, we have that for any ϵ > 0 and ν > 2 ,

Proof of Lemma E.2. The main challenge arises from the possible correlation between θ init and X 1 , . . . , X n . First note that for any ϵ > 0 , ν > 2 and a real-valued random variable Y , by Hölder's inequality, we have

Also note the useful inequality that for two real-valued random vectors (possibly correlated) V 1 , V 2 , we have

Now to control the first quantity of interest, by using a triangle inequality, we have

̸

where

and X ′ 1 is an i.i.d. copy of X 1 and in particular independent of θ init . ∆ 1 is controlled via (15):

In ( a ) , we have plugged in the definition of ∆ 1 and applied a Jensen's inequality with respect to the empirical average; in ( b ) , we have used A7 to bound the ν -th moment term as

and recalled the definitions of ϑ init n,m and ε init n,m ; ν . On the other hand,

In ( a ) , we have used that ( Kf )( x ) = ∫ K ( x, y ) f ( y ) dy ; in ( b ) , we have used that the Markov operator leaves the constant function invariant; in ( c ) , we used that K 1; θ init leaves p θ init invariant; in

( d ) , we denoted ( e l ) l ≤ d as the standard basis vectors of ❘ d and multiplied and divided by p θ init ( x ) ; in ( e ) , we have used a Cauchy-Schwarz inequality; in ( f ) , we have used the definition of the spectral gap α in A3; in ( g ) , we have used A1 and A2. Combining the bounds gives the first inequality that

Now we handle the second quantity by conditioning on θ init and perform a bias-variance decomposition:

where

Note that the covariance terms separate because X θ init i is independent of K m i ; θ init ( X i ) conditioning on θ init . η 2 Q B is exactly the quantity controlled above, so it suffices to bound the variance term Q V . By explicitly computing the second covariance term while noting that X θ init 1 , . . . , X θ init n are conditionally i.i.d. given θ init and K m i ; θ init ( X i ) 's are conditionally independent across 1 ≤ i ≤ n given A n , we have

where we have used A4, A7 and A1 in the last line. Combining the bounds, we obtain that

Proof of Lemma E.3. Since both F GD ∞ and f pop involve infinite-length Markov chains, the initializations do not matter and we can decouple the stochasticity of X i and K i ; θ init . In particular,

where

In the computations above, we have used the relation ∇ 2 θ log Z ( θ ) = Cov X ∼ p θ [ ϕ ( X )] and the assumption sup θ ∈ Ψ tr( ∇ 2 θ log Z ( θ )) = σ 2 from A1. This implies the desired bound.

Lemma E.4. Under A1, E ∥ θ pop -ψ ∗ ∥ 2 ≤ ( 1 -2 µη + L 2 η 2 ) E ∥ θ init -ψ ∗ ∥ 2 .

Proof of Lemma E.4. Recall that

By construction, f pop is deterministic and f pop ( ψ ∗ ) = ψ ∗ . By plugging in the recursions and expanding the square, we get that

In the last line, we have recalled inf ψ ∈ Ψ λ min ( ∇ 2 ψ log Z ( ψ )) = µ and sup θ ∈ Ψ λ max ( ∇ 2 ψ log Z ( ψ )) = L by A1 and applied Lemma C.8. Combining the coefficients gives the desired statement.

F Proofs for offline SGD

We prove Theorem B.1 (which directly implies Theorem 4.3) and Theorem B.2 in this section. The key ingredient of both proofs is Lemma F.1 below, which provides an iterative error bound for the SGD-with-replacement scheme by combining different approximation bounds in Appendix E. Throughout this section, we denote δ SGDw t,j := E ∥ ∥ ψ SGDw t,j -ψ ∗ ∥ ∥ 2 .

Lemma F.1. Under A1, A2, A3, A4 and A7, we have that for 1 ≤ j ≤ N -1 ,

where 1 -η t ( µ -α m σC χ -L 2 2 η t ) > 0 .

Proof of Lemma F.1. We first remark that in view of Lemma C.4, the projection step in Algorithm 2 does not increase δ SGD w t,j , so it suffices to bound δ SGDw t,j as if projection is not performed on ψ sGDw t,j . To apply the results from Appendix E, we identify θ init = ψ SGDw t,j -1 and η = η t , which allows us to write ψ SGDw t,j = θ SGDw m,B . This also implies E ∥ θ init -ψ ∗ ∥ 2 = δ SGDw t,j -1 and ε init n,m ; ν ( ϵ ) ≤ ε SGDw ν ; n,m,t ( ϵ ) .

By adding and subtracting the auxiliary gradient updates followed by expanding the square, we obtain

In ( a ) , we have expanded the square, used F n defined in Lemma E.1 and A n defined in Lemma E.2, and noted that θ pop -ψ ∗ is almost surely constant given θ init ; in ( b ) , we have applied Lemmas E.1, E.2, E.3 and E.4 under A1, A2, A3, A4 and A7, and used √ a + b ≤ √ a + √ b for a, b ≥ 0 ; in ( c ) , we have grouped the terms by powers of δ SGDw t,j -1 , bounded the indicator function from above by 1 and used B ≤ n to replace n in the denominator by B . While the computation above is complicated, we remark that the key steps are taking the conditional expectations for the crossterms in ( b ) , which gives us a tighter bound than directly applying a triangle inequality of the form E ∥ Y 1 + Y 2 ∥ 2 ≤ ( √ E ∥ Y 1 ∥ 2 + √ E ∥ Y 2 ∥ 2 ) . To further simplify the bounds, we seek to bound each

coefficient by a square, which yields

This implies

Now note that since µ ≤ L by the definitions in A1, 2 µη t -L 2 η 2 t ≤ 2 Lη t -L 2 η 2 t ≤ 1 . By using √ 1 -x ≤ 1 -x 2 for all x ≤ 1 , we get that

Substituting this into the earlier bound and taking a square-root, we obtain the desired bound that

Moreover, since L ≥ µ by definition from A1, we have

which finishes the proof.

F.1 Proof of Theorem B.1

Under A1, A2, A3, A4 and A7, Lemma F.1 implies

for 1 ≤ t ≤ T and 1 ≤ j ≤ N , where we have used ˜ µ m = µ -α m σC χ , η t = Ct -β and

By an induction on j = 1 , . . . , N , we have

By noting that δ SGDw t, 0 = δ SGDw t -1 ,N almost surely for t ≥ 2 and using another induction on t = 1 , . . . , T ,

where

Note that by Lemma F.1, we also have that 1 -˜ µ m Ct -β + L 2 C 2 2 t -2 β > 0 . This allows us to apply Lemma C.3 to obtain

To control ∑ T t =1 Q t A t t -β , recall that β ∈ [0 , 1] , ˜ µ m > 0 , L 2 C 2 2 > 0 , and that

We can now apply Lemma C.3: If β ̸∈ { 1 2 , 1 } , then

If β = 1 2 , i.e. 2 β = 1 , we have

If β = 1 , we get that

Substituting the bounds into (18), we get the desired bound that √ δ SGDw T,N is upper bounded by

F.2 Proof of Theorem B.2

Recall that δ SGDo t,j := E ∥ ∥ ψ SGDo t,j -ψ ∗ ∥ ∥ 2 . To apply the results from Appendix E to ψ SGDo t,j , we condition on S o t , which in particular fixes S o t,j , the last sizeB subset of [ n ] chosen. We then identify θ init = ψ SGDo t,j -1 , η = η t and the dataset used as D o t,j := ( X i : i ∈ S o t,j ) , which allows us to identify ψ SGDo t,j = θ GD m , the full-batch gradient descent update using D o t,j . Meanwhile, we note that almost surely θ GD m = θ SGDw m,B , the SGD-with-replacement iterate that uses the full dataset D o t,j . Observe that the proof of Lemma F.1 holds with δ SGDo t,j -1 replaced by any random initialization θ init possibly correlated with X 1 , . . . , X n , which allows us to obtain

Since the error recursion for δ SGDo t,j is identical to that of δ SGDw t,j in Lemma F.1, the proof of Theorem 4.3 follows directly, thereby yielding an identical result for δ SGDo T,N as with δ SGDw T,N in Theorem 4.3.

G Proofs for tail probability bounds in offline SGD

We present the proofs for results in Appendix B.3 that control the tail probability terms ϑ SGDw ν ; n,m,T and ε SGDw ν ; n,m,T .

Proof of Lemma B.3. For δ > 0 , let N δ be the δ -covering number of Ψ , which satisfies N δ ≤ ( r Ψ (1 + 2 /δ ) ) p (Example 5.8, [50]). Note also that by the Jensen's inequality applied to E [ · | X 1 ] and Assumption A6, there exist some σ m , ζ m > 0 such that, for any z ∈ ❘ p with ∥ z ∥ ≤ ζ m ,

Meanwhile under A5, recycling the proof of Lemma 3.1 of [21] shows that if C m δ/ √ p < σ 2 m ζ m ,

Since the probability above is an upper bound to ϑ SGDw n,m,T (3 C m δ ) , we get that if C m δ/ √ p < σ 2 m ζ m ,

Recall that by assumption, log n n < σ 2 m ζ 2 m p + ν -2 . We now choose

which implies

Taking a squareroot across and using √ a + b ≤ √ a + b for a, b > 0 gives the desired bound. The limiting result follows by substituting this bound into Theorem B.1.

Proof of Lemma B.4. Let δ > 0 and N δ be defined as in the proof of Lemma B.3, with N δ ≤ ( r Ψ (1 + 2 /δ ) ) p . Let ( ψ l ) N δ l =1 be the centers of the covering δ -balls. The covering-ball argument of the proof of Lemma 3.1 of [21] shows that

By a Markov's inequality followed by the Burkholder's inequality applied to an average of i.i.d. summands (see e.g. [51] for ν > 2 ), there exists a constant C ν > 0 depending only on ν such that

where we have used assumption A4 in the last line. This implies

Choosing δ = n -( ν -2) ν 2( ν 2 +( ν -2) p ) ≤ 1 , we get that

Taking a squareroot across and using √ a + b ≤ √ a + √ b for a, b > 0 gives the desired bound. The limiting result follows by substituting this bound into Theorem B.1.

Proof of Lemma B.5. By a Markov's inequality and a Jensen's inequality with respect to the empirical average, we have

Meanwhile by a triangle inequality and the ergodicity assumption, we have

This implies ϑ SGDw n,m,T ( ϵ ) ≤ 2 ˜ C K ˜ α m ϵ -1 , and therefore

Choosing ϵ = ˜ α ( ν -2) m/ (3 ν -2) gives

Taking a squareroot across and using √ a + b ≤ √ a + √ b for a, b > 0 gives the desired bound. The limiting result follows by substituting this bound into Theorem B.1.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract summarizes the theoretical results of our paper. We cleary mention that while CD can achieve these rates, assumptions are needed to guarantee this. We also mention the model class studied in this paper, namely (unnormalized) exponential family distributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We mention limitations multiple times in the main body of our work: after introducing the assumptions in Section 3.1, we mention that spectral gaps may not be verified for heavy tail distributions, or act numerically unfavorably for multimodal distributions, and that constants may be large in practice. In the discussion (Section 6), we reflect back on these limitations, and we mention the limitations of our considered model class (unnormalized exponential families).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All of our theoretical results are formally proved in the supplementary material. In the main paper, we provide some proof sketches (discussion of the tail term in Section 4, paragraphs after theorems in section 3) to provide high level intuitions behind the proofs. All of the theoretical results provided by our work are proved in the supplementary material, usually by invoking (and referencing) more atomic lemmas, also proved in the supplementary material. All of our theorems and lemmas are stated with the assumptions they require.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments, and thus no code or data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: As a theory paper, we believe that our work does not breach the code of conduct. With this work, we aim to advance the understanding of statistically efficient algorithms, a domain which has the potential to improve the overall computational efficiency of machine learning algorithms.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We believe that our work will not have the negative impacts mentioned (e.g., disinformation, surveillance, fairness, privacy, security considerations). We believe that advancing the domain of statistical effiency has the potential to improve the overall computational efficiency of machine learning algorithms, which, all else left equal, could constitute a positive societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Since our work does not include experiments nor data, and only analyzes existing models, we do not believe that our work requires any specific guidelines.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our work does not release any assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a non-asymptotic analysis of the Contrastive Divergence (CD) algorithms, demonstrating their ability to achieve parametric convergence rates and near-optimal asymptotic variance in the online setting. It also shows near-parametric rates in the offline setting, extending previous results. The analysis is applicable to unnormalized exponential families, which, despite their flexibility, cover only log-densities with linear relationships on the model parameters. \nThe analysis also covers various data batching schemes, including fully online and minibatch methods.\nThe study highlights the importance of extending these results to more general models in future work. Additionally, the assumption of $\\chi^2$-regularity of the exponential family model is crucial for mapping Markov contractions back to the parameter space. Future research could consider alternative conditions on the Markov Chain Monte Carlo (MCMC) chains, such as geometric ergodicity, to avoid the presence of potentially large regularity constants.\n\nSTRENGTHS:\n**Originality**: The paper provides a novel non-asymptotic analysis of the Contrastive Divergence (CD) algorithm, significantly extending the understanding of its convergence properties. Unlike prior work that focused on asymptotic convergence rates, this study shows that CD can achieve the parametric rate $O(n^{-1/2})$, which is a substantial improvement over the previously established rate of $O(n^{-1/3})$. This originality is further enhanced by applying the analysis to various data batching schemes, including online and minibatch settings.\n\n**Quality**: The theoretical contributions are robust and well-supported by rigorous mathematical analysis. The paper meticulously outlines the assumptions required for their results, such as $\\chi^2$-regularity, and provides clear justifications for these choices. The comprehensive approach to evaluating the performance of CD in different batching schemes adds depth to the quality of the research.\n\n**Clarity**: The paper is clearly written and well-structured, making it accessible to a broad audience, including those who may not be deeply familiar with the nuances of CD or unnormalized models. The authors effectively explain complex concepts and provide a logical flow from the introduction to the conclusion. The distinction between the contributions of this work and previous research is clearly articulated, enhancing the reader's understanding of the paper's significance.\n\n**Significance**: The findings have significant implications for the field of machine learning, particularly in the training of unnormalized models. By demonstrating that CD can achieve near-optimal asymptotic variance and parametric convergence rates, the paper provides valuable insights that can influence future research and practical applications. The potential to extend these results to more general models opens up new avenues for exploration and underscores the broader impact of this work.\n\nOverall, the paper makes a substantial contribution to the theoretical understanding of the CD algorithm, providing new insights that are both original and significant. The high quality of the analysis and the clarity of the presentation further enhance the impact of the research.\n\nWEAKNESSES:\nWhile I acknowledge that I am not an expert in this specific area, I have some suggestions that could improve the paper towards its stated goals. Although I did not thoroughly check the proofs due to the extensive 30-page appendix, here are my detailed comments:\n\n**Experimental Validation**: While the primary focus of this paper is theoretical, it would be beneficial to include an intuitive demonstration of the $O(n^{-1/2})$ rate, if possible. This could help bridge the gap between theory and practical understanding, making the results more accessible and relatable to a broader audience.\n\n**Clarity and Structure**: The paper currently presents a substantial amount of theoretical content, which can be overwhelming, especially for readers who may not be deeply familiar with the notation and concepts used. Highlighting the new techniques that significantly improve upon previous work would make the contributions clearer.\nConsider focusing on one main theorem with detailed lemmas and techniques that build towards it. This approach could provide a more structured and digestible presentation, making it easier for readers to follow the logical progression of the work.\n\n**Assumptions and Practicality**: The paper requires a slightly stronger moment assumption (Assumption A4) to achieve better results. Providing more detailed examples and explanations that demonstrate the practical acceptability of this assumption would be valuable. This could help convince readers that the stronger assumption is reasonable and applicable in real-world scenarios.\n\nI welcome further discussion on these points to ensure a comprehensive evaluation. Thanks!\n\nQUESTIONS:\nSee the weaknesses part.\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations in conclusion part. The paper is theoretical, and I believe there is no need to confirm societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the contrastive divergence algorithm for learning exponential family distributions.\nBoth online and offline versions are studied. The main contribution of this paper is to establish the parametric convergence rate $O(n^{-1/2})$ of the estimator, which improves the previously best result of $O(n^{-1/3})$.\n\nSTRENGTHS:\nEfficiently learning unnormalized model, especially exponential family distributions, is an important problem. \nAnalyzing the CD algorithm is known to be difficult, and this paper seems to establish the best result, proving the parametric rate both in online and offline settings.\nI like the gentle introduction of the CD in the first three pages.\n\nWEAKNESSES:\n- That said, the paper is extremely technical, loaded with heavy notation. Though I am aware of some related works on learning unnormalized models, this paper is hard to appreciate. It is hard to penetrate without extreme familiarity with the previous analysis in the literature and all the optimization analysis tools. I guess this paper could be certainly of some audience, but it'd be great if the authors can provide some more high-level explanations, pointing out some difficulties in the analysis, required techniques, and so on.\n- In particular, assumptions are not with concrete examples. Can you provide a few examples of exponential family distributions that satisfy and don't satisfy each assumption?\n\nQUESTIONS:\n- Is it proper to say \"near-optimality\", if this refers to the parametric rate $O(n^{-1/2})$, and it indeed achieves CRLB up to a factor of 4? It's still not efficient due to the multiplicative factor. Is this dependence usually called \"nearly optimal\"? For example, compared to the NCE estimators, can you argue that what's guaranteed with the CD estimator is provably better in a certain regime? This kind of insights would be very useful.\n\nLIMITATIONS:\nThere is no specific limitation noted in the paper.\nIt'd be very helpful if the authors can provide a comparison of assumptions and attained rates from previous papers to the ones in the paper, side-by-side.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies parameter estimation problem in naturally-parametrized exponential families using the contrastive divergence methods, i.e., SGD where the stochastic gradients are estimated via MCMC. Both online and offline variants of the algorithms are considered. Building upon existing works on stochastic approximation, the paper establishes upper bounds on the last iterate and averaged iterate for online algorithms. In the offline setting, the paper establishes a near-parametric rate under certain tail assumptions.\n\nSTRENGTHS:\nContrastive divergence algorithms have been used in practice, and it is interesting to provide some theoretical guarantees. The paper made solid contribution by proving sharp rate of convergence under various settings. The results provide methodological guidance on the choice of tuning parameters such as stepsizes and Monte Carlo trajectory length. The analysis in the online setting is clean and conceptually straightforward, while the results are strong.\n\nWEAKNESSES:\nOn the technical side, the results are mostly applying existing techniques directly. The online results are mostly from Bach and Moulines (2011), with a careful analysis of the biases.\n\nI'm not totally convinced by the usefulness of the offline algorithm. It seems that the analysis is much more involved and requries additional technical assumptions. On the other hand, the results are weaker than the online setting.\n\nQUESTIONS:\n- Unlike Bach and Moulines (2011), the claim of matching Cram\\'{e}r--Rao lower bound in Theorem 3.3 involves an additional factor of 2. I think this is unavoidable because of the additional variance from the Monte Carlo sample. This needs to be discussed more thoroughly.\n- The writing of Section 4 needs some improvement. The notations are extremely heavy, making the main message difficult to follow. I would suggest simplifying the expression under optimal choice of parameters, and present the main result clearly. The heavy notations can be introduced in the proof sections.\nMinor comments:\n- Eq (5). Definition of $\\mu$ is incorrect. Strong convexity parameter should be the minimum eigenvalue, instead of the operator norm.\n\nLIMITATIONS:\nThe paper addressed limitations of the results adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper delivers a detailed non-asymptotic analysis of the Contrastive Divergence (CD) algorithm applied to unnormalized exponential family distributions. It significantly advances the understanding of CD by showing that, under certain regularity conditions, the algorithm can achieve the parametric convergence rate of \\(O(n^{-1/2})\\). This finding represents a notable improvement over the previously established convergence rate of \\(O(n^{-1/3})\\). The paper's analysis is rooted in a meticulous exploration of how CD's bias and variance components affect its convergence behavior, providing deeper insights into the algorithm's performance under various conditions.\n\nSTRENGTHS:\n- The paper offers a thorough theoretical analysis, providing new insights into the convergence behaviors of CD, particularly highlighting how CD approaches the true data distribution parameters under different data batching schemes, including fully online and minibatch settings.\n- By establishing that CD's asymptotic variance is close to the Cramér-Rao lower bound, the paper positions CD as a near-optimal method for training unnormalized models, which are prevalent in fields such as image and text processing, protein modeling, and neuroscience.\n- The introduction of new primal-dual treeplex norms and a novel regret lower bound specifically tailored for sequence-form strategy spaces in online learning scenarios represent significant theoretical advancements.\n\nWEAKNESSES:\nThe paper focuses heavily on theoretical analysis with less emphasis on empirical validation. Including comprehensive experiments that benchmark CD against other estimation techniques across various real-world datasets could strengthen the claims.\n\nThe paper focuses heavily on theoretical analysis with less emphasis on empirical validation. Including comprehensive experiments that benchmark CD against other estimation techniques across various real-world datasets could strengthen the claims.\n\nQUESTIONS:\nCould you provide more detailed explanations or examples of the \"specific regularity conditions\" under which CD achieves the improved convergence rate of $O(n^{-1/2})$?\n\nAre there plans to complement the theoretical findings with empirical validations?\n\nLIMITATIONS:\nThe authors listed the limitations and future directions in their discussion section", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of finding a maximum-likelihood estimator for an exponential family. The paper proves guarantees for the Contrastive Divergence (CD) algorithm, which iteratively performs stochastic gradient descent on the cross-entropy loss between the estimated model and empirical data distribution. However, each step of gradient descent is performed by estimating the gradient with an MCMC algorithm.\n\nThe paper proves:\n* Thm 3.2: Online CD (i.e., using a fresh sample on each step), with enough MCMC steps, leads to $O(n^{-1/2})$ error rate with $n$ samples\n* Thm 3.3: Online CD with Polyak-Ruppert averaging achieves (up to a constant factor of <= 2), an asymptotically optimal rate.\n* Thm 4.2: Offline CD (i.e., using a batch of samples at each step) leads to $O((\\log(n) / n)^{1/2} + 1/\\sqrt{B})$ error rate with batch size B.\n\nSTRENGTHS:\nThis paper improves on the results of Jiang et al. [21], which had previously shown a weaker rate under stronger assumptions. The math that I checked appears correct, and the paper is very well written and presented. Many variations of CD (online, online with averaging, and offline) are analyzed. The best rates possible are also proved.\n\nWEAKNESSES:\nMy main concern is on whether the subject of this paper will be of sufficient interest to a large enough portion of the NeurIPS audience, or whether a different venue/journal with a more statistical audience might be more appropriate. The \nresults of this paper are technical bounds on how CD converges for maximum-likelihood estimation in exponential families; and the CD algorithm is not (currently) a focus of the machine learning community. Furthermore, the rate improvement proved is relatively modest -- from the previously known $O(n^{-1/3})$ rate in Jiang et al. [21] to the current best possible $O(n^{-1/2})$ rate, so to some extent the qualitative nature of the result was previously known.\n\nVery minor:\n* Assumption A.4, which is needed for Theorem 4.2 is stronger than what Jiang et al. [21] assumes (see Remark 4.1). But Theorem 4.2 is stronger than what is proved in Jiang et al. [21].\n* The bound in Line 23 of Theorem 3.2 is quite elaborate and difficult to check/use. However, the most important case of $\\\\beta = 1$ is conveniently solved.\n* On line 95, the idiom “silver lining” is used incorrectly. It should indicate something positive, rather than something negative.\n\nQUESTIONS:\n* What is $\\\\tilde{\\\\mu}$ in Theorem 3.2? Do you mean $\\\\tilde{\\\\mu}_m$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-22"}