A Computationally Efficient Method for Learning Exponential Family Distributions

Abstract: Abhin Shah, Devavrat Shah, Gregory Wornell

Reviews and author responses:

 We thank Reviewer nf2S for their detailed feedback and comments.

We respond to the main review below:

1. *Dimension $p$ of the vector of $x$*: We do not assume that $p$ is a constant. We think of $k_1$ and $k_2$ as implicit functions of $p$ i.e., we let $k_1$ and $k_2$ scale with $p$. Typically, for an exponential family, the quantity of interest is the number of parameters i.e., $k_1 \times k_2$ and this quantity scales polynomially in the dimension $p$. For example, $k_1 \times k_2 = O(p^2)$ for Ising model, $k_1 \times k_2 = O(p^t)$ for $t-$wise Markov Random Fields over binary alphabets, etc. Therefore, in this scenario, the dependence of the bounds on $p$ would also be $O(poly(p/\alpha))$.

2. *Running time for computing the gradient of $L_n(\Theta)$*: The centering of the natural statistics i.e., computing $\Phi(x) - E_{x \sim U_{\mathcal{X}}}[\Phi(x)]$ depends on the natural statistics themselves. For example, for the commonly used natural statistics such as polynomials* or sinusoidals, centering should be relatively straightforward (since the integrals/expectations would have closed-form expressions). Therefore, computing the gradient of $L_n(\Theta)$ would still be polynomial in the dimension $p$.

We will refine the exposition in the revised paper to make these points clearer.

*Polynomial natural statistics provides a comprehensive model owing to the Weierstrass approximation theorem.

This paper studies learning the natural parameter of a $k$-parameter minimal exponential family in the classical setting where the algorithm receives i.i.d. samples from the underlying distribution.

The author provides a computationally and statistically learning algorithm that outputs an estimate of the parameter of a $k$-parameter minimal exponential family within error $\alpha$ in $\ell_2$ norm with sample complexity $O(\mathrm{poly}(k/\alpha))$ and computationally complexity $O(\mathrm{poly}(k/\alpha))$ under some additional assumptions.  The innovative part of the proof is that instead directly analyzing the log-likelihood function of the minimal exponential family (which is NP-hard to compute in general), the author applies the projected gradient descent framework for smooth functions to a new loss function $L_n(\Theta)$ (which is described in Definition 3.1). In addition, the author establishes some connections between their methods and the MLE of a re-parameterized distribution.

It seems that the author assumes that the dimension $p$ of the vector $x$ is a constant and it takes $O(k_1k_2n)$ time to compute the gradient of the loss function $L_n(\Theta)$. Perhaps one thing I would like to see accounted for is the dependence of the bounds on $p$. Is the algorithm (specifically the running time for computing the gradient of $L_n(\Theta)$) still computationally efficient on the dimension $p$? This gets quite hard to track through the proofs. I think the authors adequately addressed the limitations and potential negative societal impact of their work.


 We thank the reviewer for taking out the time to go over our responses! We will incorporate the reviewer's suggestions in the revised paper.

The paper studies the problem of learning truncated minimal exponential families. Here truncated refers to bounding the distribution in a certain domain which is equivalent to a re-normalization, and minimal refers to to a non-reducible representation of the exponential family. Towards this goal, the paper proposes a computationally tractable estimator which satisfies consistency, normality and gets finite sample guarantees. They further show that the proposed estimator is closely related to the MLE of a shifted distribution.

The paper provides guarantees which cover general norms and continuous/bounded features. The proposed estimator is closely related to prior estimators (see 2 below) and has nice connections to the MLE and satisfies most desirable properties. The techniques are standard, where the loss is proven to be smooth and strongly convex (restricted) to guarantee recovery of the parameters.   Overall, the paper is well-written and covers various settings of parameter sparsity and low-rank. However, the bounds presented currently have sample complexity dependent polynomially in the parameter dimension. In most works in the field, sparsity and other assumptions allow to get away with logarithmic dependence on the parameter dimension and this is a major challenge that these works address. The same trade-off is not present here despite assuming norm-bounds on the parameters. This makes me less excited about the result. There are also a few other questions/comments I have below, addressing which would increase my confidence in the paper.

Comments/Suggestions:
1. There has been a lot of work relaxing the assumptions of node conditional learning. With only assumptions on sparsity and a lower bound on minimum weight of the underlying graph, Klivans and Meka [1] gave a computationally efficient algorithm achieving optimal rates for Ising Models/MRFs. 
2. Vuffray et. al. [2] have previously proposed an estimator ISO which is similar to the proposed objective albeit the node-wise structure. Please compare with that as well.
3. As far as I understand, in prior work on Ising models, Assumption 4.1 is in fact proven and $\lambda_{min}$ depends on minimum edge weight and $\ell_1$ norm. Since it depends on the parameter $\theta^*$ it is important to discuss why it is reasonable to make this assumption and what it entails.
4. Sparsity and low-rank structure is based on this 3-way tensor formulation. One could take alternate formulations. I believe the results shouldn't substantially change based on this. Why is the preference for such a decomposition?
5. Computing the centering of the features would require some sampling which is not described. Would be good to include.

Typos:
1. Line 266 $\phi_{max} = 2b^l$ instead of $2b^k$
2. Equation (5) and (6) should have inner products since $\phi$ aren't scalars.

[1] Klivans, Adam, and Raghu Meka. "Learning graphical models using multiplicative weights." In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS), pp. 343-354. IEEE, 2017.

[2] Vuffray, Marc, Sidhant Misra, Andrey Lokhov, and Michael Chertkov. "Interaction screening: Efficient and sample-optimal learning of Ising models." Advances in Neural Information Processing Systems 29 (2016): 2595-2603. Yes. The authors could add more discussion about the practicality of their algorithm and assumptions.

 I would like to thank the authors for their detailed response. I agree that global sparsity constraints do not directly give you a $\log n$ kind of guarantee. My concern was more so that assuming such a norm bound (local sparsity), would your results see this tradeoff. As for Assumption 4.1, it would be good to include certain examples of when it holds true. 

I will be increasing my score. Please include a discussion on the related work on sparse Ising/MRF models in the updated version along with the discussion included regarding Assumption 4.1 and the practicality of your algorithm. Experiments would significantly help with the latter.

 We thank Reviewer VbcL  for their detailed feedback and suggestions. We respond to their main review below.

1. We thank the reviewer for bringing to our attention the recent work by Dagan et al [STOC 21] for Ising models. As pointed out by the reviewer, this paper is highly relevant to our work and we will include it in our references.

2. We agree with the reviewer that an empirical study of the proposed method will help understand its practicality and contrast it with related methods. Building on our current work, we plan to conduct a detailed empirical exploration in subsequent work. 

 We thank Reviewer xzAW for their detailed feedback and suggestions. We respond to their main review below:

1. We thank the reviewer for mentioning the paper ''Interaction Screening: Efficient and Sample-Optimal Learning of Ising Model'' by Vuffrey et al. We acknowledge that the initial work of the loss function began in the aforementioned paper and we will include it in our references.

2. We agree with the reviewer that it is possible to perform empirical evaluation of the proposed method to understand its practicality and contrast it with related methods. Building on our current work, we plan to conduct a detailed empirical exploration in subsequent work. 

 We thank Reviewer xSou for their detailed feedback and suggestions. 

We respond to the main review below:

The polynomial dependence of the sample complexity on the parameter dimension is indeed a distinguishing characteristic. To the best of our knowledge, the sample complexity depends logarithmically on the parameter dimension only when the parameters are assumed to be node-wise-sparse (e.g., when the parameters are associated with a _sparse_ graphical model). In other words, these prior works require a local structure on the parameters. In contrast, our work can handle global structures on the parameters (e.g., a low-rank constraint) i.e., we focus on general exponential families beyond sparse graphical models. We are not aware of any prior work that can handle such global structures with sample complexity depending logarithmically on the parameter dimension. We would appreciate further feedback on this. Further, our method also encompasses structures like global sparsity-plus-low-rank constraints on the parameter unlike these prior works that have logarithmic dependence of the parameter dimension on the sample complexity.

We now respond to comments / suggestions below:

1. We thank the reviewer for mentioning the paper ''Learning graphical models using multiplicative weights'' by Klivans and Meka. We would like to highlight that their algorithm (called the Sparsitron) is useful only when the  parameters are assumed to be node-wise-sparse i.e., when the parameters are associated with a _sparse_ graphical model. For such _specific_ assumptions on the parameters, there has been a lot of work to relax the assumptions required for learning. However, our work focuses on global structures associated with the parameters. We leave the question of relaxing the assumptions required for learning as an open question. (Aside : Sparsitron is applicable only to sparse pairwise graphical models over general discrete alphabets and sparse t-wise graphical models over binary alphabets i.e., it is not applicable to sparse t-wise graphical models over general discrete alphabets.)

2. We thank the reviewer for mentioning the paper ''Interaction Screening: Efficient and Sample-Optimal Learning of Ising Model'' by Vuffrey et al. The initial work of the loss function in [51] (i.e., GISO) began in the aforementioned paper and the GISO can be viewed as a generalization of the ISO. More specfically, the GISO was desgined for sparse t-wise graphical models over general discrete alphabets while the ISO focused on sparse pairwise graphical models over binary alphabets (i.e., the Ising model). One cruicial difference between these loss functions (ISO and GISO) and our loss function is that the former are tailored for node-wise parameter estimation i.e., they require the parameters to be node-wise-sparse (as is the case with _sparse_ graphical models). In other words, these loss functions (ISO and GISO) are less useful when the parameters have a global structure (e.g., the low-rank constraint). On the contrary, our loss function is designed to accommodate global structures on the parameters. We will add a note to this effect in the revision.

3. We agree that, for Ising models (and related _sparse_ continuous graphical models), Assumption 4.1 is proven (e.g., Appendix T.1 of [40] provides one such analysis). However, such analysis typically requires the assumption that the parameters are _node-wise sparse_ i.e., they either require (a) a bound on the infinity norm of the parameter and a bound on the degree of each node or (b) a bound on the $\ell_1$ norm of the parameters associated with each node. In short, Assumption 4.1 is proven only for _sparse_ graphical models. The focus of our work is beyond the exponential families associated with sparse graphical models to general exponential families. We view Assumption 4.1 as an adequate condition to rule out certain singular distributions (It can be seen in the proof of Proposition E.1 that this condition effectively lower bounds the variance of a non-constant random variable). We will add a note to this effect in the revision.

4. We agree with the reviewer that alternate formulations would not change the analysis substantially. We prefer a 3-way tensor decomposition for notational ease. One could also consider $k_3$ different matrices of dimension $k_1 \times k_2$ instead of a tensor of dimension $k_1 \times k_2 \times k_3$.

5. We thank the reviewer for this suggestion. The centering of the natural statistics depends on the natural statistics themselves. For example, if the natural statistics are polynomials or sinusoidals, centering them should be relatively straightforward (since the integrals would have closed-form expressions). In other cases, one might require some sampling as mentioned by the reviewer. We will add a note to this effect in the revision.

We now respond to the review regarding the limitations ---

We thank the reviewer for this suggestion. We discuss the practicality of our algorithm and assumptions below.

* A. We believe our algorithm (Algorithm 1) is practical since the optimization in (11) is a convex minimization problem (i.e., minimizing a convex function over a convex set). There are various off-the-shelf efficient implementations of projected gradient descent algorithm that can be used. Also, computing the gradient of the loss function has a simple closed form expression (see (23) in the supplementary). Finally, our algorithm is efficient since the computational complexity is polynoimal in the parameter dimension and the error tolerance. 

* B. We make the following assumptions in our work - Assumption 2.1, 2.2, 2.3, and 4.1. 
		
* B1.  We look at Assumption 2.1 as a potential flexibility in the problem specification. In other words, a practitioner has the option to choose from a variety of constraints on the natural parameters (that could be handled by our framework). For example, in some real-world applications the parameters are sparse while in some other real-world applications the parameters have a low-rank and a practitioner could choose either depending on the application at hand.  
		
* B2. We believe that for polynomial and/or sinusoidal natural statistics, Assumptions 2.2 and 2.3 would hold whenever the domain of the random vector $\mathbf{x}$ is appropriately bounded. (Aside: Most of the previous work work only with polynoimal natural statistics). We provide a variety of examples in section 2.1 that satisfy Assumptions 2.1, 2.2, and 2.3.
		
* B3. As mentioned previously, Assumption 4.1 can be thought of as an adequate condition to rule out certain singular distributions. Therefore, we expect this assumption to hold for most real-world applications.

* B (continued) We will add these discussions in the revision as suggested.

We now respond to the review regarding typos ---

All the typographical bugs raised by the reviewer will also be corrected.

The authors consider the problem of learning exponential families with truncated support under low rank  ($O(\log n)$ rank) assumptions on the parameters. They develop a loss function following related results on structured learning. They utilize a projected gradient descent algorithm to optimize this loss function in polynomial time. They establish sample complexity for parameter recovery by optimizing this loss function.  Overall I found the paper theoretically interesting and easy to read.

The authors develop a seemingly practical algorithm under reasonable assumptions for learning exponential families. The authors clearly explain their algorithm and results and how they compare theoretically to related works. Additionally they explain the intuition of the approach well in Theorem 4.1 and its proof.

The authors extent what I believe is being called the interactive screening objective. The authors cite [51] as inspiration for the loss function, however I believe the initial work on this loss function began in Vuffrey et al., "Interaction Screening: Efficient and Sample-Optimal Learning of Ising Models", NeurIPS 2016, which is not cited.

I believe that it is possible to perform experiments for the method proposed. These experiments can be used to show how practical the method is and how it compares to related approaches. No issues.

In this paper, the authors look at the general problem of learning an exponential family of distributions from samples. They propose a new estimator which is computationally tractable. They show that the usual MLE estimator for this problem is not tractable. Finally, they show that their estimator can be seen as the $\Theta$ minimizing the KL divergence from the uniform distribution to the MLE estimator corresponding to the difference $\Theta*-\Theta$, where $\Theta*$ is the true parameter value. They show that their estimator is consistent and asymptotically normal. Unlike the MLE estimator, their estimator is not asymptotically efficient. Designing an estimator with both computational and asymptotic efficiency is left as an open problem. They also show finite sample bounds for their estimator, making certain technical assumptions that make things work. The finite bounds rely on strong convexity so that the gradient descent algorithm gets close to the true parameters.  The recent work by Dagan et al [STOC 21] for ising models, has a similar story, where the authors use a pseudo-likelihood as opposed to the MLE for reasons of computational efficiency, and show finite sample bounds using strong convexity. I think this paper is highly relevant to the current work and worth citing. 

I think an empirical study would have made the paper much stronger. I think this is a good paper and vote for acceptance. Yes

Meta-review: This paper studies the problem of learning the natural parameters of a k-parameter (minimal) exponential family, given i.i.d. samples from the underlying distribution. The main result is a computationally and statistically efficient (polynomial in k/eps) 
learning algorithm that outputs an estimate of the parameters within error eps under some additional assumptions. Instead of directly analyzing the log-likelihood of the minimal exponential family (which is computationally hard in general), this work uses projected gradient descent applied to a new loss function. A  caveat pointed out by the reviewers (and acknowledged by the authors) is that computing the gradient of the objective is not necessarily in polynomial-time, unless additional assumptions are made on the distribution. Efficient computability might be true for specific cases, but it is not true in the full generality stated in the paper. While the reviewers leaned towards accepting the paper, they believe that this point needs to be addressed in the final version.

Decision: accept