Title: Learning Exponential Families from Truncated Samples
Venue: NeurIPS 2023
Accepted: yes
Keywords: truncated statistics

robustness

exponential families

extrapolation
Abstract: Missing data problems have many manifestations across many scientific fields. A fundamental type of missing data problem arises when samples are \textit{truncated}, i.e., samples that lie in a subset of the support are not observed. Statistical estimation from truncated samples is a classical problem in statistics which dates back to Galton, Pearson, and Fisher. A recent line of work provides the first efficient estimation algorithms for the parameters of a Gaussian distribution and for linear regression with Gaussian noise.

In this paper we generalize these results to log-concave exponential families. We provide an estimation algorithm that shows that \textit{extrapolation} is possible for a much larger class of distributions while it maintains a polynomial sample and time complexity on average. Our algorithm is based on Projected Stochastic Gradient Descent and is not only applicable in a more general setting but is also simpler and more efficient than recent algorithms. Our work also has interesting implications for learning general log-concave distributions and sampling given only access to truncated data.

Learning Exponential Families from Truncated Samples

Abstract

Missing data problems have many manifestations across many scientific fields. A fundamental type of missing data problem arises when samples are truncated , i.e., samples that lie in a subset of the support are not observed. Statistical estimation from truncated samples is a classical problem in statistics which dates back to Galton, Pearson, and Fisher. A recent line of work provides the first efficient estimation algorithms for the parameters of a Gaussian distribution [10] and for linear regression with Gaussian noise [11, 14, 37].

In this paper we generalize these results to log-concave exponential families. We provide an estimation algorithm that shows that extrapolation is possible for a much larger class of distributions while it maintains a polynomial sample and time complexity on average. Our algorithm is based on Projected Stochastic Gradient Descent and is not only applicable in a more general setting but is also simpler than the recent algorithms of [10, 26, 11, 14, 37]. Our work also has interesting implications for learning general log-concave distributions and sampling given only access to truncated data.

1 Introduction

In many statistical estimation and inference problems, we have access to only a limited part of the data that would be necessary for the classical statistical methods to work, which motivates the development of statistical methods that are resilient to missing data [29]. Truncation [32, 8] is a fundamental and frequent type of missing data and arises when samples that lie outside a subset of the support are not observed and their count is also not observed. Statistical estimation from truncated samples is the focus of the field of truncated statistics, which was developed since the beginning of the twentieth century starting with the work of Galton [19], Pearson and Lee [34, 35], and Fisher [16]. Truncated statistics is widely applicable in Econometrics and many other theoretical and applied fields [32].

A recent line of work establishes the first sample optimal and computationally efficient methods for fundamental statistical estimation problems from truncated samples [10, 26, 11, 13, 14, 24, 37]. All the aforementioned works though heavily rely on the Gaussianity of the distribution of data or the Gaussianity of the noise in regression problems. Gaussianity is an idealized assumption and the question of generalizing truncated statistics beyond Gaussianity has been explored in many existing works, e.g., [1, 22, 38]. The only results in this regime though are for single dimensional problems and truncations that can be described as intervals.

0.10

TruncationSet

0.08

X2)

2.00

0.06

p(X1,

1.75

0.04

1.50

0.02

1.25

1.00

4.0

0.75

3.5

3.0

0.50

2.5

0

2

1.5X2

2.0

0.25

4

X1

6

8

10

0.0

0.5

1.0

0.00

0.000.25 0.500.751.001.25 1.501.752.00

X1

In this work we provide statistically and computationally efficient methods for estimating the parameters of exponential families from truncated samples . Our results generalize the recent work of [10] and are the first to provide an estimation algorithm for this problem for a general class of exponential families and for a general class of truncation biases.

Exponential families are one of the most influential type of distribution classes since they include many fundamental distributions such as normal, exponential, beta, gamma, chi-squared, and Weibull distributions. They were first introduced by Fisher [17] and later generalized by [9, 27, 36]. The estimation of the parameters of exponential families over continuous domains is the subject of many classical and recent results; starting from the work of Fisher [17] until the recent results of [25, 41]. This line of work has also found applications in many areas of statistics including causal inference [40]. Our work contributes in this line of research as well since we show how to estimate exponential families even when we only have access to truncated samples.

1.1 Our Results

For every distribution p , we define the truncated version p S as the distribution with density that satisfies p S ( x ) ∝ 1 { x ∈ S }· p ( x ) . Weconsider distributions p θ with support X ⊆ R m parameterized by a vector of parameters θ ∈ R k that can be written in the following form:

where h and T are known functions and T is called the sufficient statistics of the exponential family. Our goal is to estimate the vector of parameters θ up to ϵ error in ℓ 2 distance using only samples from p S θ when we have oracle access to S , i.e., there is an oracle that for every x ∈ X can answer whether x ∈ S or not.

Without any further assumptions estimation from truncated samples is impossible as shown in [10]. In this paper we use the following assumptions to get our estimation guarantees:

Assumption I: We assume a lower bound and an upper bound on the variance of the sufficient statistics in any direction. This assumption is used in estimation of the parameters of exponential families even without truncated samples, e.g., Assumption 4.1 in [41].

Assumption II: We assume that the exponential families contain log-concave distributions. We use this assumption in two places: (1) to show our extrapolation result, our distributions must satisfy an anti-concentration property; this is a property that is heavily utilized even in estimating a Gaussian distribution [10], (2) we require access to a sampling oracle for the underlying distributions; this sampling oracle can be implemented efficiently using Langevin dynamics if we have log-concavity.

Assumption III: We assume that the sufficient statistics T are bounded degree polynomials. This assumption combined with log-concavity provides the anti-concentration property that we

need. Alternatively, we can assume that the sufficient statistics belongs to a function class that satisfies anti-concentration but for simplicity of exposition we focus on the case where T ( x ) is a polynomial.

Our main result is the following.

Informal Theorem 1.1 (See Theorem 3.1 for the formal statement) . Under Assumptions I, II, III, and given samples n (at least ˜ O ( k/ϵ 2 ) for some ϵ ) from p S θ ∗ , where S is a measurable set to which we have oracle access, there exists an estimation algorithm with (expected) running time poly( m,k,n ) that outputs ̂ θ such that with probability at least 99%, it holds that ∥ ̂ θ -θ ∗ ∥ < ϵ .

Our main result has the following important implications:

▷ We show that our assumptions are satisfied from exponential distributions, Weibull distributions, continuous Bernoulli, continuous Poisson, Gaussian distributions, and generalized linear models. Hence, our result implies an efficient method for estimation from truncated samples for all these distribution classes.

▷ Another interesting corollary of our result is that we can combine it with the ideas of [12] and get a general method for learning log-concave distributions from truncated samples. In particular, assume that we want to learn from truncated samples a distribution that can be written in the form p ( x ) ∝ exp( -f ( x )) where f is a convex function. Now under mild assumptions, we can replace f ( x ) with a finite Taylor approximation, i.e., we have that f ( x ) ≊ ∑ i a i t i ( x ) for some polynomials t i ( x ) . Then, using our method we can estimate the parameters a i and output an estimation of p .

▷ In the context of sampling, the gradient of the log-likelihood is the score function that is needed to run Langevin dynamics (e.g., see [30, 7]). Our result also says that we are able to sample from the original distribution, given that we only observe truncated samples.

Technical Contributions. Our algorithm is projected stochastic gradient descent (PSGD) on the negative log-likelihood function and is the same algorithm that is used in many of the recent works in truncated statistics. As in the previous work though, the main challenge is to show that PSGD will converge in our general setting. The previous analysis of the convergence of PSGD was based on exact properties of the Gaussian distribution. When we move away from Gaussianity every step of the convergence analysis becomes much more technical and we need to make sure to only use properties that generalize to exponential families beyond Gaussian distributions. In particular, certain results such as that of Lemma 3.6 and its related quantities in C.1 can be generalized even beyond exponential families, holding for any density.

1.2 Related Work

Our most related literature is the recent series of works on truncated statistics which includes the following results: estimation of multivariate normal distributions [10], linear regression with Gaussian noise [10, 26, 11, 13, 14, 37], estimation of product distributions over the hypercube [18], non-parametric density estimation [12]. All of these works heavily rely on properties of the Gaussian distributions, or product distributions over the hypercube, or their dependence in the number of dimensions is not efficient, e.g., [12]. In our work we identify the properties of exponential families that are only required to get the efficient estimation results and we show that linear dependence on the dimension is achievable in settings that are more general than the Gaussian case.

Another related work is that of [30] that solves parameter estimation of a truncated density given samples through the score matching technique. To derive a tractable objective, we need appropriate boundary conditions which are not satisfied by truncated densities, but [30] instead uses a modified weighted Fisher distance given that the truncation set S is a Lipschitz domain (a type of open and connected set). On the other hand, our work assumes no particular structure about S and hence our results are more general and applicable in much more complicated settings for exponential families.

2 Preliminaries

Notation. Lowercase bold letters will denote real-valued vectors, e.g., x ∈ R m , and uppercase bold letters will denote matrices with real values, e.g., A ∈ R n × m . For a random vector x ∼ ρ ,

Cov [ x ] = Cov [ x , x ] = E [( x -E [ x ])( x -E [ x ]) ⊤ ] is its covariance matrix, and Var ( x ) is the trace of the covariance matrix (a scalar value). Depending on whether it is clear from context, Cov and Var may include subscripts to indicate the distribution ρ . The notation B ( c, R ) is the Euclidean ball centered at c with radius R > 0 .

Exponential Families. Let x ∈ X ⊆ R m . We are interested in a class of densities which have the form,

where h : R m ↦→ R + is the base or carrier measure , θ ∈ Θ with Θ = { θ ∈ R k : A ( θ ) < ∞} is the natural parameter space , T : R m ↦→ R k is the sufficient statistic for θ , and A ( θ ) = log Z ( θ ) is the log-partition function, where Z ( θ ) = ∫ p θ ( x ) d x .

A regular exponential family is one where Θ is an open set. It is minimal if the θ and T ( x ) are each linearly independent. Any non-minimal family can be made minimal by appropriate reparametrization. In any regular exponential family, A ( θ ) is convex. It is strictly convex if the representation is minimal. Exponential families have several nice properties (e.g., see Theorem 1 of [4]), among which are that ∇ A ( θ ) = E p θ [ T ( x )] and ∇ 2 A ( θ ) = Cov p θ [ T ( x )] .

Truncated Distributions. Let ρ be a probability distribution on R m . We represent ρ as a probability density function with respect to the Lebesgue measure d x on R m . Let S ⊆ R m be such that ρ ( S ) = α for some α ∈ (0 , 1] . Let ρ S := ρ ( · | · ∈ S ) be the conditional distribution of x ∼ ρ given that x ∈ S . Concretely, the density of ρ S is

For exponential families, we have the truncated density p S θ ( x ) is:

See Figure 1 for an illustration.

Sub-Exponential Distributions. Although the term sub-exponential has been overloaded (e.g., [21] v.s.[44]), the definition we will use describes a class of distributions whose tails decay at least as fast as an exponential, but with potentially heavier tails than Gaussians [44].

There are several equivalent characterizations of sub-exponential random variables (e.g., see Prop. 2.7.1 of [44]), one of which uses the moment generating function.

Definition (Sub-exponential random variable) . A centered, real-valued random variable X ∈ SE ( ν 2 , β ) is sub-exponential with parameters ν 2 , β > 0 if

Membership Oracle of a Set. Let S ⊆ R m . A membership oracle is an efficient procedure which computes { x ∈ S } .

✶ 3 Projected Stochastic Gradient Descent Algorithm

Problem Setup. We are given truncated samples { x i } n i =1 , with each x i ∼ p S θ ∗ , where p θ ∗ ( S ) = α > 0 . Without knowledge of the truncation set S beyond access to a membership oracle, can one recover θ ∗ and thus p θ ∗ efficiently?

We answer this question positively, under the following assumptions:

Assumption A1 (Strong Convexity, Smoothness of Non-truncated Negative Log-Likelihood over Θ ) .

for some λ, L > 0 . Here, we've abused notation for Θ which can be a subset of the entire natural parameter space. As mentioned earlier, this is always at least convex for exponential families and strictly convex in minimal representation. Thus the negative log-likelihood (of the non-truncated density) can be made strongly convex and smooth by restricting the natural parameter space appropriately.

Assumption A2 (Log-Concave Density) . The density p θ ( x ) is log-concave in x .

Assumption A3 (Sufficient Statistics T ( x ) is polynomial in x ) . T ( x ) ∈ R k has components which are polynomial in x , with degree at most d .

Assumptions A2 and A3 allow us to use the anti-concentration result needed for Lemma 3.2 which is heavily utilized even in the Gaussian case. While A2 also allows for efficient sampling via Langevin dynamics, the latter is only used in Lemma 3.2. Refer back to 1.1 for discussion of these assumptions.

Main Result.

Theorem 3.1 (Main) . Given membership oracle access to a measurable set S whose measure is some constant α ∈ (0 , 1] under an unknown exponential family distribution p θ ∗ which satisfies A1, A2, A3, and given samples x 1 , . . . , x n from p θ ∗ that are truncated to this set, there exists an expected polynomial-time algorithm that recovers an estimate ̂ θ . That is, for any ϵ > 0 the algorithm

Uses an expected ˜ O ( k/ϵ 2 ) truncated samples and queries to the membership oracle,

Runs in expected poly ( m,k, 1 /ϵ ) time.

Produces an estimate ̂ θ such that with probability at least 99%,

In order the solve this problem, we need to define an objective whose optimum is θ ∗ and we need to be able to recover it uniquely. To use maximum likelihood estimation (or minimize the negative log-likelihood), we have to be able to compute gradients which depend on the truncation set S , which we cannot do directly without more knowledge about S . However, we can sample unbiased estimates of the gradient, as long we have non-trivial mass on S at a current parameter estimate (otherwise the truncated likelihood function at that parameter is not well-defined and rejection sampling would take infinite time). To address all of these issues, the organization of this section is as follows:

Section 3.1 establishes that after truncation, the negative log-likelihood remains strongly convex and smooth (in θ ) over a subset of parameters which have non-trivial mass on the truncation set.

In Section 3.3, we show that while we do not know the truncation set, we can solve the non-truncated MLE problem with truncated samples to find an initial parameter θ 0 which assigns non-trivial mass to the truncation set.

Then given this θ 0 , in Section 3.2 we show that we can construct a set of parameters K which all assign non-trivial mass to the truncation set (and contains the true parameter θ ∗ ).

In Section 3.4, we use results from the previous sections to prove that we can efficiently recover the true parameter θ ∗ using a stochastic gradient descent procedure minimizing the truncated negative log-likelihood, which projects to the parameter space K .

3.1 Strong Convexity and Smoothness of Truncated Negative Log-Likelihood

Without truncation, recovering the true parameter θ ∗ for any parameterized distribution given samples is a classical problem solved by maximizing the likelihood (or minimizing its negation). Here, we state the main objective we will minimize through a stochastic gradient descent procedure as well as the properties of this objective that will allow us to recover θ ∗ . Define:

Note that since the Hessian is a covariance matrix which is at least PSD, this objective is always convex in θ . Thus θ ∗ is a minimizer since it satisfies the first-order optimality condition. (These calculations can be found in Appendix A.1.) However, if the objective is too flat, we may not be able to recover θ ∗ even after sufficiently reducing the objective value. For this, we prove that if the original non-truncated covariance has bounded eigenvalues, the truncated one does as well under A1, A2, and A3 at parameters which assign non-trivial mass to S .

Lemma 3.2 (Preservation of Strong Convexity under Truncation) . Assume the lower bound in A1, A2, A3. If p θ ( S ) > 0 , then

where C is a universal constant guaranteed by Theorem 8 of [5] and d is the maximum degree of T ( x ) . See proof in Appendix A.2 which follows that of [10].

Lemma3.3 (Preservation of Smoothness under Truncation) . Assume the upper bound in A1. Suppose p θ ( S ) > 0 , then

See proof in Appendix A.3. The proof is simple and can be done similarly to the previous lemma.

Thus, we have shown that as long as we optimize over a parameter space where every θ assigns non-trivial mass to the truncation set, our objective is both strongly convex and smooth. The following sections will help us determine and then construct this set given samples.

Remark. Note that the upper bound increased and the lower bound decreased for the eigenvalues of the truncated covariance matrix. Even in a one-dimensional simple case like N (0 , 1) , it is easy to construct examples of both increasing the shrinking the variance given freedom to place mass anywhere under N (0 , 1) . Thus, it may be natural that the eigenvalue range expands after truncation. Remark. Lemma 3.2 and the log-likelihood calculations are direct generalizations of prior work in the Gaussian case, where we can recover the results of [10] by noting that the re-parameterization of Gaussian parameters ( µ , Σ ) as ν = Σ -1 µ and T = Σ -1 is the natural parameterization of multivariate Gaussian distributions in exponential family form (up to constants). The sufficient statistics here T ( x ) ∝ [ x , xx ⊤ ] has components which are polynomial in x with degree at most 2, and plugging in d = 2 to Lemma 3.2 recovers Lemma 4 in [10]. Appendix B includes other examples beyond Gaussians which satisfy A1, A2, and A3.

3.2 Parameter Space with Non-Trivial Mass on Truncation Set

The prior section established that the strong convexity and smoothness parameter of the truncated objective is controlled by the mass that p θ assigns to the truncation set S for any given θ . In this section, we will prove lower bounds on the mass that p θ assigns to the truncation set, given that the parameter distance to the optimum ∥ θ -θ ∗ ∥ is bounded.

Lemma 3.4 (Lower bound for mass on truncation set under smoothness given parameter distance) . Assume A1. Let θ , θ ′ ∈ Θ . Then for two distributions from the same exponential family

Proof is provided in Appendix C.3, and only needs smoothness. Thus, we can lower bound the mass that a parameter θ assigns to S given its distance ∥ θ -θ ∗ ∥ from θ ∗ which is assumed to have p θ ∗ ( S ) = α .

Thus, to make use of this property we want to establish a procedure to initialize a parameter θ 0 such that its distance to the optimum θ ∗ is bounded. Then during the optimization procedure we will make sure to make progress toward θ ∗ . The following is the result we will be able to prove after proving some results between truncated and non-truncated quantities in the following Section 3.3.

Corollary 3.5 (Parameter space with non-trivial mass on truncation set) . Given θ 0 such that E p θ 0 [ T ( x )] = T where T = 1 n ∑ n i =1 T ( x i ) is the empirical mean sufficient statistics given our samples { x i } n i =1 for each x i ∼ p S θ ∗ , if we define

then p θ ( S ) ≥ α 2 exp ( -6 L λ 2 ( ϵ S + O (log 1 /α )) 2 ) > 0 holds ∀ θ ∈ K (with probability at least 1 -δ for n ≥ Ω(log(1 /δ )) ). Furthermore, θ ∗ ∈ K (as long as θ ∗ ∈ Θ satisfies conditions of Claim 1).

This result will follow from Lemma 3.4, Claim 1, Lemma 3.7, and Lemma 3.9 in the next section.

3.3 Initialization with Empirical Samples and Non-truncated MLE

Given samples from the truncated density p S θ ∗ , one may first try to solve the non-truncated empirical MLE problem to find a parameter θ 0 without truncation and hope that it is good enough. (We will show that it is.) In order to understand how good this initial guess is, we need to establish some relationships between the truncated and non-truncated density.

Lemma 3.6 (Truncated vs. Non-truncated Mean Sufficient Statistics for General Densities) . Let ρ be a probability distribution on R d (not necessarily from an exponential family). Let S ⊆ R d with ρ ( S ) > 0 . Then

Proof of this lemma and several related quantities for general truncated densities is in Appendix C.1. In low dimensions, this variance term may effectively be a constant; however, in high-dimensional settings this term can grow with dimension (which is undesirable if we want an efficient algorithm). Given more assumptions about the density, we can get better dimension-free bounds which generalize the results from the Gaussian case.

Claim 1. Let θ ∈ Θ such that θ + 1 β u ∈ Θ for some β > 0 for all unit vectors u and such that Assumption A1 holds for p θ from an exponential family. Then X := u ⊤ ( T ( x ) -E x ∼ p θ [ T ( x )]) is SE ( L, β ) . (Proof provided in Appendix C.2.)

Remark. This claim allows us to have concentration of the empirical mean sufficient statistics to its population mean for the non-truncated distribution. The prior work [10] analyzed the mean and covariance of a Gaussian separately, but for instance to establish bounds on distances between the truncated and non-truncated mean parameter, it made use of Gaussian concentration inequalities (which are tighter than sub-exponential ones). The relationship between the truncated density and the non-truncated one will allow us to say that the empirical truncated mean sufficient statistics is also somewhat close to the non-truncated population mean.

Lemma 3.7 (Concentration of Empirical vs. Non-truncated Mean Sufficient Statistics) . Suppose θ ∗ satisfies the conditions of Claim 1 and p θ ∗ ( S ) = α > 0 . Let T = 1 n ∑ n i =1 T ( x i ) be the empirical mean sufficient statistics given our samples { x i } n i =1 each x i ∼ p S θ ∗ . Let ϵ S > 0 . For n ≥ Ω ( 2 β ϵ S log ( 1 δ ) ) , with probability at least 1 -δ ,

It should be noted that it suffices to consider ϵ S as some independent constant (and not related to the ϵ accuracy parameter in the main theorem). However, by taking n at least ˜ O ( k/ϵ 2 ) as stated in the main theorem, this will be small. See proof in Appendix C.4.

Remark. At a high level, the truncated samples can be thought of as O ( n/α ) samples from the non-truncated distribution (keeping only those in S ), and each are 'not too far' (depending on how much mass the set S has under the non-truncated distribution) from the non-truncated mean due to concentration. Note that the O (log 1 /α ) term will never disappear even as we increase n , which quantifies the inherent bias that the truncated mean sufficient statistics will have with respect to the non-truncated one (and is large if the mass α is small). From this, we can also say something about the population mean sufficient statistics, one on the truncated distribution and one on the non-truncated.

Corollary 3.8 (Truncated vs. Non-truncated Mean Sufficient Statistics) . Let θ satisfy the conditions of Claim 1 and p θ ( S ) > 0 . Then

The proof follows from the preceding lemma, replacing α with p θ ( S ) and taking n →∞ . Compare this to the Gaussian case [10], where the mean and truncated means were bounded as ∥ µ -µ S ∥ ≤ O ( √ log 1 /p θ ( S )) and separately the covariances were bounded as ∥ Σ -1 / 2 Σ S Σ -1 / 2 -I ∥ F ≤ O (log 1 /p θ ( S )) . Note the smaller O ( √ log 1 /p θ ( S )) quantity due to tighter Gaussian concentration vs. the sub-exponential rate.

Once we have bounds on the norm of the difference between the truncated and non-truncated mean sufficient statistics, we can bound distance in parameter space. The following completes this.

Lemma 3.9 (Non-truncated MLE Solution Distance to θ ∗ ) . Suppose θ ∗ satisfies the conditions of Claim 1 and p θ ∗ ( S ) = α > 0 . Let θ 0 be such that E p θ 0 [ T ( x )] = T where T = 1 n ∑ n i =1 T ( x i ) given each x i ∼ p S θ ∗ . Let ϵ S > 0 and n > Ω ( 2 β ϵ S log(1 /δ ) ) . Then w.p. at least 1 -δ ,

Proof. Define ℓ untr ( θ ) := E x ∼ p θ 0 [ -log p θ ( x )] . Its gradient and Hessian calculations can be done similarly to ℓ ( θ ) , the truncated version, but with S = X the full support of the distribution.

Since E z ∼ p θ ∗ [ T ( z )] -E x ∼ p θ 0 [ T ( x )] = ∇ ℓ ( θ ∗ ) untr is the gradient of the untruncated negative log-likelihood whose optimum is at θ 0 , by A1 this gives

where the result follows from the fact that E z ∼ p θ 0 [ T ( z )] = T and ∥ E z ∼ p θ ∗ [ T ( z )] -T ∥ ≤ O (log 1 /α + ϵ S ) w.p. 1 -δ from Lemma 3.7.

Note that this result combined with Lemma 3.4 gives Corollary 3.9.

3.4 Analysis of Projected Stochastic Gradient Descent Algorithm

Now we have all the tools we need to analyze the main algorithm. For ease of notation, define d ( α ) := ϵ S + O (log 1 /α ) which is a constant that depends on α . The following describes the projected stochastic gradient descent algorithm referenced by Theorem 3.1.

Algorithm 1 Projected SGD Algorithm Given Truncated Samples

Algorithm 2 SampleGradient

Given the results from the previous sections, we can now prove the main result. The analysis is based on that of Chapter 5 (Theorem 5.7) of [20] which we modify and state below:

Theorem 3.10 (SGD Convergence) . Let f be a λ -strongly convex function. Let θ ∗ ∈ arg min θ ∈ K f ( θ ) . Consider the sequence { θ t } N t =1 generated by SGD (Algorithm 3) and { v t } N t =1 the sequence random vectors satisfying E [ v t | θ t ] = ∇ f ( θ t ) and E [ ∥ v t ∥ 2 | θ t ] < ρ 2 for all t , with a constant step size η satisfying 0 < η < 1 λ . It follows that for t ≥ 0 ,

The proof is adapted and reproduced in Appendix D.1 for completeness. To apply the above theorem we need to take care of statistical problems:

(i) strong convexity f is a strongly convex function over K a convex set

(ii) smoothness f is a Lipschitz-smooth function over K

(iii) feasibility of optimal solution θ ∗ ∈ K .

(iv) bounded variance step for all t , E [ ∥ v t ∥ 2 | θ t ] < ρ 2 for some ρ 2

and algorithmic ones:

(a) initial feasible point efficiently compute an initial feasible point θ 0

(b) unbiased gradient estimation efficiently sample an unbiased estimate of ∇ f (= ∇ ℓ )

(c) efficient projection efficiently project to parameter space K

Statistical problems. Firstly, (iii) is assumed. (i-ii) is addressed by Lemmas 3.2, 3.3, 3.6, 3.9, 3.7, 3.4. In particular, we can initialize with θ 0 such that ∥ θ 0 -θ ∗ ∥ ≤ d ( α ) λ by Lemmas 3.6, 3.9, 3.7, with probability at least 1 -δ . Given this θ 0 , we can construct K = B ( θ 0 , d ( α ) λ ) ∩ Θ which has the property that

Thus by Lemma 3.4, we will also have

where κ = L/λ is the condition number. Since we are projecting to K in which all parameters have non-trivial mass, our objective remains strongly convex. In particular, our objective ℓ ( θ ) has

where λ S = 1 2 ( α 2 exp ( -6 κ λ · ( d ( α )) 2 ) 4 C deg ) 2 deg λ and L S = exp ( 6 κ λ · ( d ( α )) 2 ) α 2 L are some constants which depend on α , λ , L , and the maximum degree, deg, of the sufficient statistics. It remains to address (iv), which is done by the following lemma.

Lemma 3.11 (Bounded variance step) . Let v i denote the output of SampleGradient ( x i , θ i ) at any iteration i ∈ [ N ] . Provided that ∥ E S p θ [ T ( x )] -E p θ [ T ( x )] ∥ ≤ O (log 1 /p θ ( S )) for all θ ∈ K ,

Proof. At any iteration i (arbitrary),

In the last step, we've used the fact that

by assumption, smoothness, Cor. 3.8 and Lemma 3.9.

Algorithmic problems. For the algorithmic problems, by Cor. 3.8 and Lemmas 3.9, 3.7, we can address (a) by solving the empirical MLE problem with no truncation. Given that we can efficiently sample exactly (or approximately see Appendix D.2) from the non-truncated p θ for any θ , we can sample unbiased gradients via Algorithm 2 with expected O (1 /p θ i ( S )) = O ( exp(6 κ · ( d ( α )) 2 ) α 2 ) samples at each step t to address (b). Point (c) can be done efficiently, since our parameter space is a simple intersection of Euclidean balls if we choose Θ to be a Euclidean ball that sits inside the whole parameter space which contains θ ∗ .

Let D ( k, L, λ, α ) = k ( L S + L )+(1+2 κ ) 2 (6 κ ( d ( α )) 2 -2 log α ) 2 λ 2 S ϵ 2 . Putting everything together, to get E ∥ θ N -θ ∗ ∥ 2 ≤ ϵ 2 , the number of iterations and samples should be

provided that η = min { λ S ϵ 2 2 ρ 2 , 1 λ S } , applying Lemma D.1 to the bound from Theorem 3.10 with A = ρ 2 λ S , C = λ S , µ = 2 λ S , and ρ 2 = kL S + kL +(1 + 2 κ ) 2 ( O (log 1 /p θ ( S ))) 2 by Lemma 3.11. Further, Lemma 3.4 guarantees O (log 1 /p θ ( S )) ≤ O ( log exp(6 κ · ( d ( α )) 2 ) α 2 ) for all θ ∈ K .

In probability, we get P ( ∥ θ N -θ ∗ ∥ 2 ≥ 3 ϵ 2 ) ≤ 1 / 3 by Markov's inequality. Then we can amplify the probability of success to 1 -δ by repeating the procedure from scratch log 1 /δ times, as in [10]. Given a polynomial time (poly ( m,k, 1 /ϵ ) ) algorithm A S to sample from p θ for all θ , each iteration takes expected O (1 /p θ i ( S )) = O ( exp(6 κ λ · ( d ( α )) 2 ) α 2 ) times the running time of A S plus the projection step (which is also efficient). This completes the result of Theorem 3.1.

Remark. In the Gaussian case, the sample complexity was given in terms of m , the dimension of the x and was stated in [10] as ˜ O ( m 2 /ϵ 2 ) . For multivariate Gaussian, the dimension k of θ is the dimension of [ x , xx ⊤ ] (vectorized) which is m + m 2 , thus we can recover the previous result.

Numerical example. While a lot of the analyses shown can seem complicated, they give some guarantees for a rather simple algorithm: simply initialize your parameters using MLE as if there is no truncation at all, then run projected stochastic gradient descent using rejection sampling. We do not even need to describe the truncation set S as long as we can query whether a point is inside S or not efficiently. As a proof of concept for the use of this algorithm, we've implemented our simple projected SGD algorithm to learn the parameters of multivariate exponential distributions, given truncated samples. To save space in the main body for exposition of the theoretical results, we've included the results and details in Appendix E.

4 Discussion

To our knowledge, this work is the first which develops a computationally and statistically efficient algorithm for learning from samples truncated to very general sets S of the form in [10] in high dimensions whose distribution does not rely on Gaussianity. This work also has interesting implications for learning general log-concave distributions through applying the Taylor theorem to the log density as in [12] and for sampling (e.g., as in [30]). Through generalizing the previous Gaussian results to general exponential families, we also extract the broader abstract properties (e.g., concentration and anti-concentration) of distributions for which the previously proposed projected stochastic gradient descent procedure still applies. It would be interesting to understand how much these results can be generalized to densities beyond exponential families, or even to extend the current work presented from expected polynomial running time to deterministic polynomial running time. We hope that our work provides the foundation for future work in this direction.

References

MABeg. Optimal tests and estimators for truncated exponential families. Metrika , 29(1):103113, 1982.

Daniel Bernoulli. Essai d'une nouvelle analyse de la mortalité causée par la petite vérole, et des avantages de l'inoculation pour la prévenir. Histoire de l'Acad., Roy. Sci.(Paris) avec Mem , pages 1-45, 1760.

Sébastien Bubeck, Ronen Eldan, and Joseph Lehec. Sampling from a log-concave distribution with projected langevin monte carlo. Discrete & Computational Geometry , 59, 06 2018.

Róbert Busa-Fekete, Dimitris Fotakis, Balázs Szörényi, and Manolis Zampetakis. Optimal learning of mallows block model. ArXiv , abs/1906.01009, 2019.

A Carbery and J Wright. Distributional and l-q norm inequalities for polynomials over convex bodies in r-n. Mathematical research letters , 8(3):233-248, May 2001.

Yeshwanth Cherapanamjeri, Constantinos Daskalakis, Andrew Ilyas, and Manolis Zampetakis. What makes a good fisherman? linear regression under self-selection bias, 2022.

Sinho Chewi. Log-concave sampling. Book draft available at https://chewisinho. github. io , 2022.

A Clifford Cohen. Truncated and censored samples: theory and applications . CRC press, 2016.

Georges Darmois. Sur les lois de probabilitéa estimation exhaustive. CR Acad. Sci. Paris , 260(1265):85, 1935.

C. Daskalakis, T. Gouleakis, C. Tzamos, and M. Zampetakis. Efficient statistics, in high dimensions, from truncated samples. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS) , pages 639-649, Los Alamitos, CA, USA, oct 2018. IEEE Computer Society.

Constantinos Daskalakis, Themis Gouleakis, Christos Tzamos, and Manolis Zampetakis. Computationally and statistically efficient truncated regression. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 955-960. PMLR, 25-28 Jun 2019.

Constantinos Daskalakis, Vasilis Kontonis, Christos Tzamos, and Emmanouil Zampetakis. A statistical taylor theorem and extrapolation of truncated densities. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 1395-1398. PMLR, 15-19 Aug 2021.

Constantinos Daskalakis, Dhruv Rohatgi, and Emmanouil Zampetakis. Truncated linear regression in high dimensions. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 10338-10347. Curran Associates, Inc., 2020.

Constantinos Daskalakis, Patroklos Stefanou, Rui Yao, and Emmanouil Zampetakis. Efficient truncated linear regression with unknown noise variance. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 1952-1963. Curran Associates, Inc., 2021.

Raaz Dwivedi, Yuansi Chen, Martin J. Wainwright, and Bin Yu. Log-concave sampling: Metropolis-hastings algorithms are fast. Journal of Machine Learning Research , 20(183):1-42, 2019.

RA Fisher. Properties and applications of Hh functions. Mathematical tables , 1:815-852, 1931.

Ronald Aylmer Fisher. Two new properties of mathematical likelihood. Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character , 144(852):285-307, 1934.

Dimitris Fotakis, Alkis Kalavasis, and Christos Tzamos. Efficient parameter estimation of truncated boolean product distributions. In Conference on Learning Theory , pages 1586-1600. PMLR, 2020.

Francis Galton. An examination into the registered speeds of american trotting horses, with remarks on their value as hereditary data. Proceedings of the Royal Society of London , 62(379387):310-315, 1897.

Guillaume Garrigos and Robert M. Gower. Handbook of convergence theorems for (stochastic) gradient methods, 2023.

Charles M. Goldie and Claudia Klüppelberg. Subexponential Distributions , page 435-459. Birkhauser Boston Inc., USA, 1998.

Patrick M Hannon and Ram C Dahiya. Estimation of parameters for the truncated exponential distribution. Communications in Statistics-Theory and Methods , 28(11):2591-2612, 1999.

Andrii Ilienko. Continuous counterparts of poisson and binomial distributions and their properties. Annales Universitatis Scientiarum Budapestinensis de Rolando Eötvös Nominatae. Sectio Computatorica , 39, 03 2013.

Andrew Ilyas, Emmanouil Zampetakis, and Constantinos Daskalakis. A theoretical and practical framework for regression and classification from truncated samples. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 4463-4473. PMLR, 26-28 Aug 2020.

Sham Kakade, Ohad Shamir, Karthik Sindharan, and Ambuj Tewari. Learning exponential families in high-dimensions: Strong convexity and sparsity. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , volume 9 of Proceedings of Machine Learning Research , pages 381-388, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR.

V. Kontonis, C. Tzamos, and M. Zampetakis. Efficient truncated statistics with unknown truncation. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS) , pages 1578-1595, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society.

Bernard Osgood Koopman. On distributions admitting a sufficient statistic. Transactions of the American Mathematical society , 39(3):399-409, 1936.

Alice Lee. Table of the Gaussian 'Tail' Functions; When the 'Tail' is Larger than the Body. Biometrika , 10(2/3):208-214, 1914.

Roderick JA Little and Donald B Rubin. Statistical analysis with missing data , volume 793. John Wiley & Sons, 2019.

Song Liu, Takafumi Kanamori, and Daniel J. Williams. Estimating density models with truncation boundaries using score matching. Journal of Machine Learning Research , 23(186):138, 2022.

Gabriel Loaiza-Ganem and John P. Cunningham. The continuous bernoulli: fixing a pervasive error in variational autoencoders. ArXiv , abs/1907.06845, 2019.

Gangadharrao S Maddala. Limited-dependent and qualitative variables in econometrics . Number 3. Cambridge university press, 1986.

Oren Mangoubi and Nisheeth Vishnoi. Sampling from log-concave distributions with infinitydistance guarantees. Advances in Neural Information Processing Systems , 35:12633-12646, 2022.

Karl Pearson. On the systematic fitting of frequency curves. Biometrika , 2:2-7, 1902.

Karl Pearson and Alice Lee. On the generalised probable error in multiple normal correlation. Biometrika , 6(1):59-68, 1908.

Edwin James George Pitman. Sufficient statistics and intrinsic accuracy. In Mathematical Proceedings of the cambridge Philosophical society , volume 32, pages 567-579. Cambridge University Press, 1936.

Orestis Plevrakis. Learning from censored and dependent data: The case of linear dynamics. CoRR , abs/2104.05087, 2021.

Mathias Raschke. Inference for the truncated exponential distribution. Stochastic environmental research and risk assessment , 26(1):127-138, 2012.

Adrien Saumard and Jon A. Wellner. Log-concavity and strong log-concavity: A review. Statistics Surveys , 8(none):45 - 114, 2014.

Abhin Shah, Raaz Dwivedi, Devavrat Shah, and Gregory W Wornell. On counterfactual inference with unobserved confounding. arXiv preprint arXiv:2211.08209 , 2022.

Abhin Shah, Devavrat Shah, and Gregory Wornell. A computationally efficient method for learning exponential family distributions. Advances in neural information processing systems , 34:15841-15854, 2021.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge University Press, USA, 2014.

John W Tukey. Sufficiency, truncation and selection. The Annals of Mathematical Statistics , pages 309-311, 1949.

Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.

A Proofs and Calculations Regarding the Objective

A.1 The Truncated Negative Expected Log-Likelihood Function

The negative log-likelihood that x ∈ S is a sample of p S θ ( x ) is

Its gradient w.r.t. θ is

The Hessian is

We can similarly define the population negative log-likelihood as

A.2 Proof of Lemma 3.2

Proof. Define the following quantities:

Claim 2. R ′ ⪰ R ∗ . (Proof in Appendix A.4.)

Now, let ξ ∈ R k with ∥ ξ ∥ 2 2 = 1 arbitrary. Then

where p ξ ( x ) , p ′ ξ ( x ) are polynomials of degree at most 2 d whose coefficients depend on ξ (under A3). Furthermore, note that for any ξ ∈ R k , p ξ ( x ) ≥ 0 and p ′ ξ ( x ) ≥ 0 (due to the rank one matrix inside the expectation being PSD).

First, since R ′ ⪰ R ∗ ⇐⇒ ξ ⊤ R ′ ξ ≥ ξ ⊤ R ∗ ξ , we have

Now define the set A := { x : p ′ ξ ( x ) ≤ γ } for γ = ( β 4 Cd ) 2 d λ where p θ ( S ) = β > 0 . Theorem 8 of [5] says

Now we can split E z ∼ p S θ [ p ′ ξ ( z ) ] into the part on S ∩ A and S ∩ A c . Note that if p θ ( S ) = β and p θ ( A ) ≤ β 2 , this implies p θ ( S ∩ A c ) ≥ β 2 as

Then

A.3 Proof of Lemma 3.3

Proof. Similar to the proof of the previous lemma, define the following quantities:

Claim 3. It holds that R ′′ ⪰ R . (Similar proof to Claim 2.)

Let ξ ∈ R k with ∥ ξ ∥ 2 2 = 1 arbitrary. Then

where f ξ ( x ) , f ′ ξ ( x ) are some functions which depend on x and ξ (e.g., polynomials of degree at most 2 d under A3). By the previous claim, we also have

Note that

Since λI ⪯ R ∗ ⪯ LI by A1, it holds that ξ ⊤ R ∗ ξ = E x ∼ p θ [ f ξ ( x )] ≤ L , thus the following inequalities hold:

A.4 Proof of Claim 2

We will prove a general claim which should take care of both claims in Lemmas 3.2 and 3.3.

̸

Claim 4. Let x ∼ ρ be a random vector with mean µ . Let b be another vector such that b = µ . Then

Proof.

As a corollary, since the second term is a rank-1 matrix (thus PSD), we have that E [( x -b )( x -b ) ⊤ ] ⪰ E [( x -µ )( x -µ ) ⊤ ] .

B Examples of Other Distributions which Satisfy Assumptions

Example 1 (Exponential Distribution) . The exponential distribution density can be written

defined on x ∈ R + which is a convex set and for λ > 0 . In natural form, it is

defined for θ < 0 . Note that

T ( x ) = x is a polynomial in x .

This is log-linear in x (so log-concave in x ).

Variance of the sufficient statistic is simply the variance, which is 1 /θ 2 > 0 for any θ < 0 . If we restrict θ in a bounded set, the negative log-likelihood will be strongly convex and smooth in θ .

Example 2 (Weibull Distribution with known shape k ) . The Weibull distribution with known shape k > 0 has density

defined on x ∈ R + and λ > 0 . We can re-parameterize this in terms of θ = -1 λ k with θ < 0 as

T ( x ) = x k is polynomial in x .

p θ ( x ) is log-concave in x if k > 1 (where recall x ∈ R + and θ < 0 ).

The variance of the sufficient statistic can also be found by taking the second derivative of A ( θ ) = -log k -log( -θ ) w.r.t. θ , which is also 1 /θ 2 > 0 .

Then

Example 3 (Continuous Bernoulli) . The continuous Bernoulli density [31] can be written

with support x ∈ [0 , 1] and λ ∈ (0 , 1) . We can re-parameterize this in terms of θ = log λ 1 -λ with θ ∈ [0 , ∞ ) so

T ( x ) = x is polynomial in x .

p θ ( x ) is log-linear in x (so log-concave).

The variance of sufficient statistic is simply the variance again, which is given by

This is strictly positive and bounded for all values of λ (thus all values of θ ).

Example 4 (Continuous Poisson) . A continuous version of the Poisson distribution (although there can be others [23]) can be written

with support x ∈ [0 , ∞ ) and λ ∈ (0 , ∞ ) . We can write this with θ = log λ so

T ( x ) = x is polynomial in x .

p θ ( x ) is log-concave in x for x ∈ R + .

In λ parameters, the mean of this distribution is λ through usual calculations (e.g., similar to those of the Gamma distribution). Note: we can absorb the e -λ term into the partition function.

Similarly, we should be able to show the variance is λ as usual. In θ space, this means the variance is exp( θ ) for θ ∈ R which is always positive. Again, we can make it bounded by restricting θ to some set.

Example 5 (Multivariate Gaussian) . The multivariate Gaussian also satisfies all of these properties. Recall that the sufficient statistics of the multivariate Gaussian has

T ( x ) = [ x , xx ⊤ ] is a polynomial in the components of x with degree at most 2 (where the xx ⊤ term can be thought of as the vector after standard vectorization).

The multivariate Gaussian density is strongly log-concave.

Then

Then

The covariance matrix (of the sufficient statistics) has a complicated form, which the authors of [10] have analyzed the lower bound for, e.g., in their Claims 1 and 2. As before, we can restrict our parameter space to ensure upper bounds.

Example 6 (Generalized Linear Models) . This example is the same as the one given in [25] for generalized linear models. It is restated here for completeness.

Consider when we have some covariance, response pair ( X,Y ) drawn from some distribution D . Suppose that we have a family of distributions P ( · | θ ; X ) such that, for each X , it is an exponential family with sufficient statistic t y,X

We can consider a one-dimensional exponential family q ν with parameterization ν = ⟨ θ, X ⟩ , then

where we see that t y,X = yX and the log partition function A ( θ, X ) = log Z ( ⟨ θ, X ⟩ ) . When q ν is Bernoulli family or unit variance Gaussian family, this corresponds to logistic regression or least squares regression , respectively.

We can appropriately generalize this to beyond linear models (e.g., polynomials) provided that we can keep the distribution log-concave.

Comment on A3. We mentioned in the main paper that this assumption combined with logconcavity provides the anti-concentration property that we need for Lemma 3.2. We assume it for simplicity of exposition, but it should be noted that as long as we have the type of anti-concentration property to control how much the covariance can shrink under truncation, we do not necessarily need T ( x ) to be polynomial. However, we've provided examples of exponential families which already satisfy this above (and there are potentially more which can be addressed by this framework that do not have polynomial sufficient statistics but nonetheless exhibit similar anti-concentration properties).

C Proofs Relating Truncated and Non-Truncated Quantities

C.1 General Truncated Densities

Let ρ be a probability distribution on R d . Let S ⊆ R d be such that ρ ( S ) = α for some α ∈ (0 , 1] . Let ρ S := ρ ( · | · ∈ S ) be the conditional distribution of x ∼ ρ given that x ∈ S .

Note that the relative density is

Then we can compute that the Rényi divergence is a constant for any order 1 ≤ q ≤ ∞ .

We recall the following general estimates.

Lemma C.1. For any probability distributions ρ, π (such that the quantities below are finite):

Proof. The first two claims are immediate by Cauchy-Schwarz. For the third one, recall we can write

Then by applying part (1) to ρ ⊗ 2 and ( π ) ⊗ 2 , we get

For our application, we have the following. Given a probability distribution ρ on R d , we let µ ( ρ ) = E ρ [ x ] be its mean, and for k ∈ N ,

So for example we have M 2 ( ρ ) = √ Var ρ ( x ) . We also have M k ( ρ ) ≤ M ℓ ( ρ ) if k ≤ ℓ .

Lemma C.2. Let ρ be a probability distribution on R d . Let S ⊆ R d with ρ ( S ) = α ∈ (0 , 1] . Then

In particular, if α ∈ (0 , 1] is such that 1 α 2 ≤ 1 + c 2 M 2 ( ρ ) 4 2 M 4 ( ρ ) 4 for some 0 ≤ c < 1 , then

Note that the constraint on α above implies 1 α 2 ≤ 3 2 , so α ≥ √ 2 / 3 . But if M 2 ( ρ ) ≪ M 4 ( ρ ) , then 1 -α will be very small.

Recall also that under some conditions, e.g. if ρ is log-concave, then we have the reverse bound that

for a universal constant C 2 , 4 , so the constraint above is not too restrictive, as it allows 1 -α of constant size.

C.2 Exponential Families with Strongly Convex and Smooth Log-Partition Functions are Sub-Exponential

Let θ ∈ Θ such that θ + 1 β u ∈ Θ for some β > 0 for all unit vectors u and such that Assumption A1 holds for p θ . Then X := u ⊤ ( T ( x ) -E x ∼ p θ [ T ( x )]) is SE ( L, β ) .

Proof. WLOG, consider p θ in the transformed space x ↦→ T ( x ) so that

where θ ∈ Θ and A ( θ ) = log( Z ( θ )) = log ( ∫ T ( X ) h ( t ) exp( θ ⊤ t ) d t ) is the log-partition function. Note that ∇ 2 A ( θ ) = Cov t ∼ p θ ( t ) [ t ] = Cov x ∼ p θ ( x ) [ T ( x )] , and by A1, A ( θ ) is a λ -strongly convex and L -smooth function in θ .

To show that p θ ( t ) is sub-exponential with parameters ( ν 2 , β ) we need to show that its moment generating function satisfies E [ e γ u ⊤ ( t -µ ) ] ≤ e γ 2 ν 2 / 2 , where µ = E p θ [ t ] , u is a unit vector, for | γ | < 1 /β .

The inequality we need to show is equivalent to proving

Since A ( θ ) is L -smooth, we have that

where we've used the property of exponential families that the gradient of the log partition function is the mean sufficient statistic. Now we can see that the appropriate parameter for ν 2 is L and γ must be small enough so that γu + θ ∈ Θ , i.e., | γ | < 1 β for some β > 0 . This is possible if θ is bounded away from the boundary of Θ .

Remark. In the above, we only needed to use that p θ is an exponential family distribution and that its log-partition function A ( θ ) is smooth. It is also possible to show that p θ has exponentially decreasing tails (in quantities involving x rather than T ( x ) ) if it is log-concave in x (assumption A2), e.g., by [39].

C.3 Proof of Lemma 3.4

Let p θ ( x ) = h ( x ) exp( ⟨ θ, T ( x ) ⟩ -A ( θ )) and A : Θ → R is the log-partition function:

Lemma C.3. For any q > 1 , θ , θ ′ ∈ Θ :

Proof.

Lemma C.4. Assume A is convex and L -smooth on Θ . For any S ⊆ X , and θ , θ ′ ∈ Θ :

Since A is convex and L -smooth,

Therefore,

Compare this to the Gaussian case (e.g., see H.8 of [37]) where this was p θ ( S ) ≥ α 2 exp ( -r · √ 2 log 1 /α -1 2 r 2 ) for ∥ θ -θ ′ ∥ < r .

C.4 Proof of Lemma 3.7

Let T = 1 n ∑ n i =1 T ( x i ) be the empirical mean sufficient statistics given our samples { x i } n i =1 each x i ∼ p S θ ∗ .

with probability at least 1 -δ .

Proof. Let ν ∗ = E x ∼ p θ ∗ [ T ( x )] .

For any event A , we have that

and for the product measure with n independent components P Π i ∈ [ n ] p S θ ∗ [ A ] ≤ ( 1 α ) n P Π i ∈ [ n ] p θ ∗ [ A ] . So we can bound the probability of events on p S θ ∗ with those on p θ ∗ . In particular, by Claim 1 and by the composition property of independent sub-exponential random variables, we have that

To translate this to the probability of the same event on p S θ ∗ , note that

which holds when t = 2 β ( log 1 /α + 1 n log 1 /δ ) . Thus for n > 2 β ϵ S log 1 /δ samples from the truncated p S θ ∗ we have that with probability at least 1 -δ , the quantity ∥ T -ν ∗ ∥ ≤ 2 β (log 1 /α ) + ϵ S .

D Additional Proofs for Algorithm Analysis

Algorithm 3 Stochastic Gradient Descent

Return θ T

D.1 SGD Algorithm and its Analysis

Although the setting of Theorem 5.7 of [20] is when the objective is a sum of many functions, the proof and its result can be easily adapted to our setting.

Theorem. Let f be a λ -strongly convex function. Let θ ∗ ∈ arg min θ ∈ K f ( θ ) . Consider the sequence { θ t } N t =1 generated by SGD (Algorithm 3) and { v t } N t =1 the sequence random vectors satisfying E [ v t | θ t ] = ∇ f ( θ t ) and E [ ∥ v t ∥ 2 | θ t ] < ρ 2 for all t , with a constant step size η satisfying 0 < η < 1 λ . It follows that for t ≥ 0 ,

Proof. At any iteration i ,

where the last line comes from multiplying the line (1) with the transpose of the same equation on either side. After projecting to the set K to obtain θ i +1 = arg min θ ∈ K ∥ ˜ θ i +1 -θ ∥ 2 and given that θ ∗ ∈ K , we have that ∥ ˜ θ i +1 -θ ∗ ∥ 2 ≥ ∥ θ i +1 -θ ∗ ∥ 2 , so

Fixing θ i in the i th iteration and taking the conditional expectation in (2) gives

where the last line is due to strong convexity, ⟨∇ f ( θ i ) , θ i -θ ∗ ⟩ ≥ λ ∥ θ i -θ ∗ ∥ 2 . By taking iterated expectations and recursively applying the above, we get that

where in the second line we used that ∑ T -1 i =0 (1 -2 ηλ ) i = 1 -(1 -2 ηλ ) T 1 -(1 -2 ηλ ) < 2 2 ηλ provided η < 1 /λ .

We can derive the complexity (number of iterations) to get E [ ∥ θ T -θ ∥ 2 ] < ϵ using the following Lemma from [20].

Lemma D.1 (Lemma A.2 of [20]) . Consider the recurrence given by

where µ > 0 , and A,C ≥ 0 are given constants and η < 1 /C . If

then

Note that to get bounds on ∥ θ T -θ ∗ ∥ rather than ∥ θ T -θ ∗ ∥ 2 , we can solve the number of iterations we need to get ϵ 2 on the right hand side, and we will get the number of iterations for ∥ θ T -θ ∗ ∥ < ϵ . Then the resulting complexity bounds will replace 1 /ϵ with 1 /ϵ 2 .

D.2 Approximate Sampling of Non-Truncated Distribution

Many analyses of stochastic gradient descent assume unbiased directions at every iteration of the algorithm, but since we need to be able to sample from p θ i multiple times at each iteration i until we get a sample in S , our directions are only unbiased if we can indeed sample exactly from p θ i each time for all i .

However if p θ i is complicated, exact sampling can be difficult or take too long. Since we assume that p θ is log-concave in x for all θ ∈ Θ , we can at least approximately sample from it efficiently via Langevin Monte Carlo, MALA, or other algorithms with convergence guarantees for log-concave densities.

Lemma D.2 (SGD Analysis with Biased Directions) . Let f be a λ -strongly convex function. Let θ ∗ ∈ arg min θ ∈ K ⊆ B ( θ ∗ ,D ) f ( θ ) . Consider the sequence { θ t } N t =1 generated by SGD but with { ˜ v t } N t =1 the sequence random vectors satisfying E [ ˜ v t | θ t ] = b t - ∇ f ( θ t ) with ∥ b t ∥ < B and E [ ∥ ˜ v t ∥ 2 | θ t ] < ρ ′ 2 for all t , with a constant step size η satisfying 0 < η < 1 λ . It follows that for t ≥ 0 ,

Proof. Define b t := E [ b t | θ t ] -∇ f ( θ t ) the bias for each t ≥ 0 . The analysis of Theorem 3.10 can be applied generically to get Eq. (2):

Now when taking the conditional expectation, we get

Taking iterated expectations and recursively applying this gives now

where the second line holds if ∥ b i ∥ ≤ B, ∥ θ i -θ ∗ ∥ < D, ∀ i (which holds under the assumptions).

Note that in our Algorithm 1, D here is simply 2 λ d ( α ) by construction. We can also control B through the following.

Bounding the bias. Fix some t . Let ˜ v t := T ( z ) -T ( x ) where z ∼ ˜ p S θ i and x ∼ p S θ ∗ . Then we can write

If we know that T ( x ) is bounded over S , we can upper-bound this given the TV distance between ˜ p S θ i and p S θ i :

Since we assume that E p θ i [ T ( x )] is finite, it should be the case that T ( x ) is bounded over its support except potentially on some negligible sets. In that case, we can replace T ( x ) with ˜ T ( x ) which replaces those potentially infinite values on negligible sets with 0 and the integral expression in (3) would be equal to one which uses ˜ T ( x ) instead of T ( x ) , and the bound on its norm holds given that ˜ T ( x ) is bounded.

Otherwise we can use bounds from Lemma C.1 to bound this as

if we have control over the chi-square divergence (see Section C.1 for definitions).

If we know that T ( x ) is a 1-Lipschitz, real-valued function (e.g., when T ( x ) = x ), we can use the dual representation of W 1 distance to bound this as

Proposition D.3 (Bounds on truncated total variation, given bounds on non-truncated) . Suppose ∥ ˜ p θ i -p θ i ∥ TV ≤ ϵ TV for some ϵ TV > 0 . Then

Proof. First, given that ∥ ˜ p θ i -p θ i ∥ TV ≤ ϵ TV , we have by one characterization of the total variation distance (the supremum of the difference in mass over all measurable sets)

Now for the truncated densities,

Efficient, Approximate Sampling. There exist several results in sampling which give bounds in TV distance in polynomial time (mixing time bounds) for log-concave distributions, e.g., [15], [3], [33] but which also usually require that the log density is also smooth (in x , not θ ). There are also proofs for Langevin Monte Carlo when the log density is convex and Lipschtiz, not necessarily smooth (e.g., see Chapter 4 of [7]), or under LSI (which is implied by strong log-concavity) with convergence in Renyi divergence (e.g., Chapter 5 of [7]). We can also use the proximal sampler to achieve convergence in KL divergence under log-concavity (e.g., Chapter 8.4 of [7]), which by Pinsker's inequality can bound the TV distance.

Proposition D.4 (Bounded variance step with bias) . If E [ ∥ v i ∥ | θ i ] ≤ ρ 2 where v i = T ( z ) -T ( x ) with z ∼ p p S θ i and x ∼ p S θ ∗ ,then E [ ∥ ˜ v i ∥ | θ i ] ≤ ρ ′ 2 for ˜ v i = T ( ˜ z ) -T ( x ) with ˜ z ∼ ˜ p p S θ i and x ∼ p S θ ∗ , where

Proof. As in the proof of the exact sampling version, we can write

We can bound the difference Var ˜ p S θ i ( T ( z )) -Var p S θ i ( T ( z )) if have bounds on Var ˜ p S θ i ( T ( z )) , through bounds like in Lemma C.1 if we can say something about the chi-square divergence, or through similar arguments to the bias bound if we assume some bounds on ∥ T ( x ) ∥ 2 over its support.

E Numerical Example

To illustrate how the algorithm performs in different dimensions, we implemented our algorithm for 2-, 5-, 10-, and 20-dimensional exponential distributions. In all cases, the truncation set is the (hyper-)cube [0 , 2] d . We chose true parameters in all cases which resulted in an initial error at most 2.5. In all cases, we use 1500 iterations and step size 0.01, each repeated 10 times. In the end, all have (average) L2 error at most 0.15. For stability (and to bypass repeating the algorithm multiple times as stated in the analysis), we instead calculated gradients using the average of 10 samples which was sufficient to have stable training results. See Figure 2.

2D

5D

2.0

10D

20D

1.5

Error

21.0

0.5

0.0

0

2

4

6

8

10

12

14

Iterations%1oo

The wall clock time to finish all 1500 iterations of training for 5-, 10-, and 20-dimensions was 42 . 9 ± 2 . 2 , 49 . 1 ± 6 . 5 , and 61 . 2 ± 3 . 0 seconds, respectively. We can see that the running time is not doubling with the doubling of dimensions.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a case where one has observations from a censored exponential family (i.e., an exponential family, but each distribution is conditioned to lie within a particular set) and wants to estimate the parameters of that exponential family.  An additional complication in the setup is that one does not have access to the set that observations are conditioned to lie within, but instead one has access to an oracle that can by queried and will tell you whether any given point lies within the truncation set or not.  This exact setting has been considered previously but restricted to less general families of distributions (e.g., multivariate Gaussians).  The authors develop a provably accurate estimator for the parameters of the exponential family that runs in polynomial time in all of the relevant parameters.  The main idea is to first perform maximum likelihood as if the data were not censored, and then perform projected stochastic gradient descent.  Stochastic gradient estimates can be obtained in this setting by using the oracle to perform rejection sampling using the non-truncated distribution defined by the current parameter estimate as a proposal distribution.\n\nSTRENGTHS:\n* The proofs are thorough, and, as far as I can tell, correct.\n* The authors make an admirable attempt to first discuss their assumptions and results at a non-technical level before present the more technical formal results.\n* The results are quite general, and the proposed algorithm is exceptionally simple and straightforward, making the approach appealing and elegant.\n\nWEAKNESSES:\n* The flow of the technical part of the paper was very difficult to follow.  The major presentation aspect that I felt was missing was a big picture overview of both the algorithm and the proof technique.  I felt like there was too large a disconnect between the informal statement of the results, and then diving directly into technical assumptions and lemmas without giving any intuition about why they were important or where they would be used.  I realize that space is limited in this venue, but perhaps more of the technical results could be moved to the supplement to make more room.\n* I also feel that the paper is missing some motivation.  To me, generalizing existing results is not in itself interesting to a broad audience.  I could imagine that the generalization presented in this paper is broadly useful, but some additional context would go a long way toward proving that point.  In particular, I am having trouble imagining a scenario where one has access to an oracle that can tell whether a given point would have been censored or not, but one does not have access to knowledge of the censoring set itself.  Just listing some specific, motivating real world examples would be useful.  I am not embedded in the truncated statistics community, so perhaps such motivation is obvious, but to a broader community I think additional motivation would be useful. \n* Lastly, some numerical examples checking the proof results would be useful.  How does running projected SGD on truncated data perform in some real world cases?  A comprehensive application or simulation study is obviously beyond the scope of this paper (and not necessary) but just a few toy examples would go a long way in giving an independent check of the analytical results.\n\nTypos / minor comments:\n* Line 34: \"is the first to provide\" --> \"are the first to provide\"\n* Line 76: Totally up to the authors, but it may be helpful to remind the reader that $m$ is the dimension of $\\theta$ here.\n* Line 192: \"log likelihood\" --> \"log-likelihood\"\n* Line 271: this sentence is not grammatically correct\n* Equation following Line 272: The \"$B$\" notation has not been defined.  Presumably it's an ($\\ell_2$?) ball centered at the first argument with radius the second argument.\n* In the equations around lines 458-460, it would be good to use a different dummy variable than $\\mathbf{x}$ in the integral as $\\mathbf{x}$ is already being used outside of the integral.\n* Equation after line 469: $p_\\xi^*$ is not defined, but based on the argument it matches the definition of $p_\\xi$ in the text above line 469.\n* Line 575: I believe that instead of $\\rho^S$ it should be $\\pi$.\n\nQUESTIONS:\n* The wording of assumption II is ambiguous.  Do the exponential families need to contain _exclusively_ log-concave distributions?  Seems like yes from Assumption A2.\n\n* Does assumption A4 require additional assumptions on $g(s)$ (e.g., being finite)?  It seems like one could _define_ g(s) as $\\mathbb{E}[X | X \\ge s]$ and then the condition vacuously holds.\n\n* The informal version of the main result has a dependence on sample size, but the formal version does not.  I believe that the dependence on $n$ comes from $\\epsilon$ but in the setup of the theorem, it says we are given samples $\\mathbf{x}_1, \\ldots, \\mathbf{x}_n$ which makes $n$ seem fixed.\n\nLIMITATIONS:\nThe authors do not include a broader impact statement.  I don't believe the work has any direct negative societal impacts, but it would be good for the authors to say so.  Additionally, I could imagine that estimating population parameters from censored data could, in principle, pose privacy risks, and so the authors may wish to make some minor comments along these lines.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-22"}

comments: SUMMARY:
This paper focuses on estimating a parametric family of distributions using a truncated sample, i.e.  the sample can only be collected within a subset of the data domain. The primary objective is to develop efficient algorithms for estimating the parameters of this family. To achieve this, previous findings on truncated samples from the Gaussian case are extended to exponential families.

STRENGTHS:
This is a theoretical paper that offers valuable support for the proposed algorithms. The extension to exponential families is significant.

WEAKNESSES:
This paper is solely theoretical and lacks numerical support for the proposed algorithms and theorems. It does not provide concrete examples to motivate the study. For instance, it would be helpful to provide real-world examples where the oracle property holds.

QUESTIONS:
You mentioned that "Some intermediate results can even be generalized to distributions beyond exponential families."  What are the key factors that are needed for such an extension?  Can you provide some concrete examples where such extensions are feasible?

LIMITATIONS:
I haven’t come across any stated limitations, but it’s possible that I may have overlooked them.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new general method for estimating the parameters of exponential-family (EF) distributions from truncated samples. The method is presented as an algorithm, which allow us to obtain good estimates of $\\theta$ in polynomial time and certain accuracy conditions. The whole methodology extends previous advances in parameter estimation from truncated samples (these last ones mainly with Gaussian densities and regression problems). The formal results included seem to be theoretically well grounded on the properties of exponential families, exploiting previous methods to both lower and upper bound the covariance of the sufficient statistics given the truncated samples. \n\nIn practice, the algorithm looks efficient (to me), as it seems to compute a certain gradient step given a measure of distance between the sufficient statistic functions of the truncated samples and the true samples of the formal EF distribution. Later, this parameter update is corrected (as it was only computed from a truncated sample that might guide it towards bad locations), and in particular, projected into the \"correct\" parameter space where it will finally converge to the desired estimate.\n\nSTRENGTHS:\nDespite some points and details that I will later add, I liked the paper quite a lot, in my opinion, the construction of the theory around exponential families is clear and strong. This last point is indeed not easy, as EF have very good properties that are however not that easy to exploit without the proper experience. In that sense, I also found very strong the use of such properties to build the main properties of the objective in section 3.1 (the three equations behind the first paragraph) which to me seem to open the door to the main result provided in the algorithm. Also surprisingly, the algorithm for estimation seems easy and direct to use, even if theory is a bit more complex. To me the paper has several strengths that is worth remarking: 1) the generalisation of the previous theory on estimation from truncated samples to exponential families, 2) the clarity and utility of the exponential family principles to make the parameter estimation possible and 3) the final result provided in the algorithm which makes the estimation possible.\n\nWEAKNESSES:\nIn my opinion, there are two lingering points that makes me think the paper is perhaps weak in some degree..\n\n**Point 1.** First, even if I liked the idea, the utility, the algorithm and the results provided in a significant way, I had a hard time reading it and kind of building myself the answers to what the main contributions and strengths of the work are. What I want to say is that somehow the synthesis of the work has been led to the readers, instead of building the story of the paper in a direct and more clear way. I'm not saying that the theoretical results should not be included in the main manuscripts, but some of the proofs could be perhaps moved to the appendix and glue all technical details a bit better to understand where the contribution of each one is important for the final algorithm. In this way I'm referring to the sections 3.2, 3.3 and 3.4 mainly. On the other hand I think that the presentation of the algorithm is great and it seems to be a fabulous result (at least to me), but one has to dig quite a lot to find what it the role of the previous elements in that one. This is important, particularly for a submission of this type, for reproducibility and clarity -- which in the end leads to impact, of course. \n\n**Point 2.** The second one is related to the lack of empirical results. As I said, even being a paper with relevant contributions and technical discoveries, I had the feeling once I finished to read it that I would have liked to see some empirical results. Mainly on the performance on the algorithm for different sets  of truncated samples, dimensionality and types of probability distributions. I understand the class of paper that this one is, mostly theoretical, and that once one proves such good estimates of parameters on exponential family distributions, which are super general, but to me it does not prevent the benefits that empirical results could produce on the clarity and quality of the manuscript.\n\nQUESTIONS:\n**Q1** -- Is there any sort of truncation effect on the samples that the given algorithm could not deal with properly?\n\n**Q2** -- Is there any fundamental difference between $\\rho$ and $p_\\theta$ in the Equations below L138, L139 and the probability distribution mentioned in L230?\n\n**Q3** -- In L122, the Cov[x,x] is described as a matrix, later for the Lemmas 3.2 and 3.3, the same covariance is upper and lower bounded using scalars. How this bound of a matrix with scalars work? Are we bounding all elements in the matrix?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proves the convergence of projected stochastic gradient descent (SGD) on the maximum-likelihood problem associated with estimating the parameters of an exponential family given truncated samples.\nThis result generalises previous work that shows similar results for Gaussian distributions to exponential families. To this end, the paper assumes bounded variance, log-concavity, sufficient statistics being polynomials, and bounded conditional expectation in the tails of the distributions.\nThese assumptions imply the convergence of SGD and, thereby, the existence of a polynomial-time algorithm for parameter estimation given truncated samples.\n\nSTRENGTHS:\nI am writing this review as an outsider to truncated statistics.\n\nThat said, even as an outsider, I find this submission well-organised and readable despite its technicality, which speaks for the clarity of the exposition. I feel a little less like an outsider after studying this work.\n\nThe results appear to be novel, and I have the impression that constructing an algorithm that estimates any (sufficiently well-behaved) exponential family from truncated samples (and not \"just\" a Gaussian; quotation marks because this kind of analysis requires non-trivial work even for Gaussian distributions, see [10]) is a significant result.\n\nWEAKNESSES:\nAs mentioned above, I am not an expert in truncated statistics. \n\nEven though, overall, I think this is a nice paper, I would like to comment on some potential weaknesses:\n\n1. Without having studied reference [10] in full detail, it seems that the analysis in the submitted paper mirrors that in [10] rather closely. A similarly structured analysis can be fine, but a more specific contextualisation of Section 3 may be warranted. I would like to hear what the authors think about this.\n\n\n2. Although I am fully aware that this paper is theoretical, reading about the efficiency of the proposed algorithm makes me want to see the algorithm \"in action\". Numerical demonstrations only appear to be optional in this line of work: some related works include them (e.g. [15, 31]), and some do not (e.g. [10, 12, 14]). But I think the present paper would appeal to a broader audience if it contained a few demonstrations of projected SGD for learning parameters of exponential families (maybe including, but not restricted to, Gaussians). \n\n\n3. I found the appendices really helpful in understanding this work (some contain crucial information for proving some of the results). However, it did not help the readability of this manuscript, having to jump back and forth between the appendix and the main paper. Some parts must inevitably move to the appendix to condense such a comprehensive analysis into nine pages. But this makes me wonder whether a conference with a page limit of nine pages is the appropriate venue for this work. (For context: since related work of comparable volume (e.g. [14, 15]) has been published at Neurips, there seems to be a precedence for accepting this submission despite its length).\n\nQUESTIONS:\nCovered by \"Weaknesses\" above.\n\nLIMITATIONS:\nLimitations appear to be discussed wherever appropriate.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}