Title: Online Structure Learning for Sum-Product Networks with Gaussian Leaves
Authors: Wilson Hsu
Agastya Kalra
Pascal Poupart
Venue: ICLR 2017
Keywords: Unsupervised Learning
Deep learning

Abstract: Sum-product networks have recently emerged as an attractive representation due to their dual view as a special type of deep neural network with clear semantics and a special type of probabilistic graphical model for which inference is always tractable. Those properties follow from some conditions (i.e., completeness and decomposability) that must be respected by the structure of the network.  As a result, it is not easy to specify a valid sum-product network by hand and therefore structure learning techniques are typically used in practice.  This paper describes the first {\em online} structure learning technique for continuous SPNs with Gaussian leaves. We also introduce an accompanying new parameter learning technique.

TL;DR: This paper describes the first online structure learning technique for continuous SPNs with Gaussian leaves.

Review discussion:
How are the likelihoods computed for the generative moment matching networks? Using Parzen windows? "A note on the evaluation of generative models" by Theis et al provides good evidence that these can be unreliable. Have the authors considered comparing to the NICE and Real-NVP models which have fully tractable likelihoods and model real-valued data?
Thank you for the question.  As mentioned in the paper, the results in Table 1 are from Jaini et al. 2016 (except for oSLRAU).  Jaini et al. wrote that Parzen windows were used to approximate the likelihood of the generative moment matching networks and also cautioned the reader that this approximation may not be close to the true likelihood as pointed out by Theis et al.  We will repeat this explanation in the next version of the paper.
Thank you for recommending to compare SPNs to NICE and Real-NVP. We are looking into the feasibility of doing such a comparison. Stay tuned.
We just uploaded a revised version of the paper with the requested clarification regarding the evaluation with generative moment matching networks.
Also we contacted Laurent Dinh (author of NICE and Real-NVP) who gave us links to publicly available code.  After inspection of the code, we determined that we will need to modify it for online training and dataset compatibility.  We are currently working on this.  Stay tuned.
How do the structure learning results vary as a function of the chosen correlation coefficient threshold (0.1 in experiments).  Same question for threshold (4 in manuscript) for choosing which structure technique to chose (combining distributions in multivariate leaf vs. mixture).
Thank you for asking how the results vary as a function of the parameters of the algorithm.  We will include some additional results to answer this question in a future version of the paper.
We just uploaded a revised version of the paper with additional experiments that show how the log-likelihood, training time and SPN size change as we vary the correlation threshold.
In short, when the threshold is too small, superfluous correlations tend to be detected which might lead to overfitting and when the threshold is too large, some correlations might be missed which might lead to underfitting.
We are currently working on additional experiments to illustrate the effect of the threshold for the maximum number of variables in a multivariate Gaussian leaf node.  Stay tuned.
We just uploaded a new version of the paper that shows how the results vary as a function of the correlation threshold and the maximum number of variables in each leaf.
The authors contribute an algorithm for building sum-product networks (SPNs) from data, assuming a Gaussian distribution for all dimensions of the observed data.  Due to the restricted structure of the SPN architecture, building a valid architecture that is tailored to a specific dataset is not an obvious exercise, and so structure-learning algorithms are employed.  For Gaussian distributed observations, the authors state that the previous state of the art is to chose a random SPN that satisfies the completeness and decomposibility constraints that SPNs must observe, and to then learn the parameters (as done in Jaini 2016).  In the contributed manuscript, the algorithm begins with a completely factorized model, and then by passing through the data, builds up more structure, while updating appropriate node statistics to maintain the validity of the SPN.
The above Jaini reference figures heavily into the reading of the paper because it is (to my limited knowledge) the previous work SOTA on SPNs applied to Gaussian distributed data, and also because the authors of the current manuscript compare performance to datasets studied in Jaini et al.  I personally was unfamiliar with most of these datasets, and so have no basis to judge loglikelihoods, given a particular model, as being either good or poor.
Nevertheless, the current manuscript reports results on these datasets that better (5 / 7) than other methods, such as SPNS (constructed randomly), Stacked Restricted Boltzmann Machines or Generative Moment Matching networks.
Overall:
First let me say, I am not really qualified to make a decision on the acceptance or rejection of this manuscript (yet I am forced to make just such a choice) because I am not an expert in SPNs. I was also unfamiliar with the datasets, so I had no intuitive understanding of the algorithms performance, even when viewed as a black-box.  The algorithm is presented without theoretical inspiration or justification.  These latter are by no means a bad thing, but it again gives me little hold onto when evaluating the manuscript.  The manuscript is clearly written, and to my limited knowledge novel, and their algorithm does a good job (5/7) on selected datasets.
My overall impression is that there isn't very much work here (e.g., much of the text is similar to Jaini, and most of the other experiments are repeated verbatim from Jaini), but again I may be missing something (this manuscript DOES mostly Jaini). I say this mostly because I am unfamiliar with the datasets.  Hopefully my reviewing peers will have enough background to know if the results are impressive or not, and my review should be weighted minimally.
Smallish Problems
I wanted to see nonuniform covariances in the data of the the toy task (Fig 3) for each gaussian component.
The SPN construction method has two obvious hyper parameters, it is important to see how those parameters affect the graph structure. (I submitted this as a pre-review question, to which the authors responded that they would look into this.)
Thank you for the review.  We just uploaded a revised version of the paper.
Regarding the lack of theoretical justification, we included an explanation about how we derived the parameter update algorithm, including a theorem that shows that it necessarily increases the likelihood of the last data point processed.
Regarding the similarities between this paper and Jaini et al. 's paper, the only content that was copied from Jaini are the results for alternative approaches in Table 1. This saved us the trouble of re-implementing these approaches and re-running them. This is common practice when a set of benchmarks can be reused. Note also that the background section in both papers introduces some basics about SPNs, but this is necessary for completeness. The contributions of both papers are different. Jaini et al.
introduced an online parameter update algorithm (Bayesian moment matching). There was no structure learning algorithm and therefore random structures were used. In our paper, we introduce an online structure learning algorithm. The algorithm is simple and therefore it might be tempting to write that there is not enough work for a publication, but simple is good in practice. Its simplicity is indeed what allows it to scale. We also hope that its simplicity will incite others to use it and perhaps enhance it.
Some questions that I believe should be answered in the paper:
+ sec3.1: Where does this parameter update algorithm come from? Was something similar done before? How does this compare with the previous algorithms for learning SPN parameters?
+ sec3.2: Two approaches are introduced.
(1) Are they equivalent? (no)
(2) Which one is more expressive?
(3) Which approach under which conditions?
+ sec3.2: If the children are subtrees rather than leaves, how are these subtrees combined?
The text states "it works the same way" but this is quite nontrivial to me.
+ sec4.2 why is VoxForge not included in the table?
Are there results with the other baseline methods?
Thank you for your feedback. We will be incorporating it into our update with answers to this and a lot more hopefully by tonight or tomorrow morning.
We just uploaded a revised version of the paper.  It includes some additional explanation about how we derived the parameter update algorithm, including a theorem that shows that it necessarily increases the likelihood of the last data point processed.  In the next version that we plan to upload we will add a discussion about the relationship to other parameter learning algorithms.
In Section 4.2, we added experiments with 5 datasets (in addition to VoxForge) and created several tables.  At the moment, the only baseline we have for those experiments consist of random structure SPNs (which is the state of the art). That being said we are working on including a comparison to NICE and RealNVP as suggested by another reviewer.
In Section 3.2, the two approaches to introduce a correlation (i.e., multivariate Gaussian and mixture of Gaussians) are not equivalent.  The mixture approach is more expressive since mixture of Gaussians with arbitrarily many components can approximate any distribution while a multivariate Gaussian can only represent a unimodal distribution with a specific shape.
So, in principle, we could use mixtures of Gaussians only, however the use of multivariate Gaussians in the leaves can often reduce the number of components in a mixture.  This is why we use both approaches.  If the children are subtrees rather than leaves, we use the mixture approach to combine them as described in the paper.  We will add an explanation to that effect in the next version of the paper.
# Summary
This paper proposes an algorithm to learn the structure of continuous SPNs in a single pass through the data,
basically by "growing" the SPN when two variables are correlated.
## NOTE
I am not an expert on SPNs, and can not really judge how impressive the presented results are due to lack of familiarity with the datsets.
# Pro
- This looks like possibly impactful work, proposing a simple and elegant algorithm for learning SPN structure single-pass, rather than just using random structure which has been done in other work in the online settings.
# Con
- The paper is heavily updated between submission deadline and submission of reviews.
- The paper reads like a rush job, sloppily written - at least the first version.
- Comparison to literature is severely lacking; eg "several automated structure learning techniques have been proposed" followed by 6 citations but no discussion of any of them, which one is most related, which ideas carry over from the offline setting to this online setting, etc.
Also since this work presents both joint structure & *parameter* learning, comparison to the online parameter learning papers (3 cited) would be appreciated, specifically since these prior approaches seem to be more principled with Bayesian Moment Matching in Jaini 2016 for example.
- I do not know enough about SPNs and the datasets to properly judge how strong the results are, but they seem to be a bit underwhelming on the large datasets wrt Random
# Remaining questions after the paper updates
- Table 3: Random structure as baseline ok, but how were the parameters here learned? Your simple running average or with more advanced methods?
- Table 1: you are presenting *positive* average log-likelihood values? This should be an average of log(p<=1) < 0 values? What am I missing here?
I recommend reject mostly because this paper should have been finished and polished at submission time, not at review deadline time.
Thank you for the review.  We just uploaded a new version that addresses many of the concerns raised.
"The paper is heavily updated between submission deadline and submission of reviews."
We realize that the paper has evolved substantially since the first submission.  All the revisions were done in response to requests by the reviewers.  ICLR differs from other conferences since authors are encouraged to submit revisions that address the reviewers's concerns.  This is better since the reviewers can evaluate the final version as it is instead of wondering whether the authors will fulfill the requests of the reviewers in the final version.
"Comparison to literature is severely lacking"
We added a new subsection about parameter learning in the background  section and expanded the subsection about structure learning.  The main characteristics of each algorithm are now described.  We also added some text in the section where we describe the new parameter and structure learning technique to clarify the differences with the closest algorithms.
"Table 3: Random structure as baseline ok, but how were the parameters here learned? "
The parameters are updated using the same parameter learning technique as in oSLRAU.
"Table 1: you are presenting *positive* average log-likelihood values? This should be an average of log(p<=1) < 0 values? What am I missing here?"
Log likelihood is positive when the density is higher than 1.  Recall that we are working with continuous variables and therefore we have a density function (instead of probabilities) and that a density function can have values higher than 1.
The authors present an online learning method for learning the structure of sum-product networks. The algorithm assumes Gaussian coordinate-wise marginal distributions, and learns both parameters and structure online. The parameters are updated by a recursive procedure that reweights nodes in the network that most contribute to the likelihood of the current data point. The structure learning is done by either merging independent product Gaussian nodes into multivariate leaf nodes, or creating a mixture over the two nodes when the multivariate would be too large.
The fact that the dataset is scaled to some larger datasets (in terms of the number of datapoints) is promising, although the number of variables is still quite small. Current benchmarks for tractable continuous density modeling with neural networks include the NICE and Real-NVP families of models, which can be scaled to both large numbers of datapoints and variables. Intractable methods like GAN, GenMMN, VAE have the same property.
The main issue with this work for the ICLR audience is the use of mainly a set of SPN-specific datasets that are not used in the deep learning generative modeling literature. The use of GenMMN as a baseline is also not a good choice to bridge the gap to the neural community, as its Parzen-window based likelihood evaluation is not really meaningful. Better ways to evaluate the likelihood through annealed importance sampling are discussed in "On the Quantitative Analysis of Decoder-Based Generative Models" by Wu et al. I would recommend the use of a simple VAE type model to get a lower bound on the likelihood, or something like Real-NVP.
Most neural network density models are scalable to large numbers of observations as well as instances, and it is not clear that this method scales well "horizontally" like this. Evaluating the feasibility of modeling something like MNIST would be interesting.
SPNs have the strength that not only marginal but also various type of conditional queries are tractable, but performance on this is not evaluated or compared. One interesting application could be in imputation of unknown pixels or color channels in images, for which there is not currently a high-performing tractable model.
Despite the disconnect from other ICLR generative modeling literature, the algorithm here seems simple and intuitive and convincingly works better than the previous state of the art for online SPN structure learning. I think VAE is a much better baseline for continuous data than GenMMN when attempting to compare to neural network approaches.
Further, the sum-product network could actually be combined with such deep latent variable models as an observation model or posterior, which could be a very powerful combination.
I would like it if these SPN models were better known by the ICLR probabilistic modeling community, but I do not know if this paper does enough to make them relevant. As with the other reviewers, I am not an expert on SPNs.
However, this seems to be a simple and effective algorithm for online structure induction, and the scalability aspect is something that is important in much recent work in the learning of representations. I think it is good enough for publication, although I would prefer to see many of the above additions to more clearly bridge the gap with other literature in deep generative modeling.
Thank you for your review.  We just uploaded a new version f the paper that addresses many of the concerns raised.  We included an empirical comparison with RealNVP and we explained that the algorithm scales quadratically with the number of features.
We note the request for more comparisons to more types of generative deep neural networks.
However, it is important to not loose track of the main contribution of the paper: an online structure learning algorithm for SPNs with Gaussian leaves. By comparing SPNs to other generative models, we are not comparing different structure learning techniques, but simply different models.
In fact, no structure learning technique has been proposed for generative models other than SPNs (as far as we know).
We also not the insistence on scalability with respect to the number of features.  As mentioned above, oSLRAU scales quadratically with respect to the number of features.
While this does not permit to scale to large images with many pixels, please remember that this is the first online structure learning technique for SPNs with Gaussian leaves while other types of networks do not have any online structure learning technique. A fair comparison would be to compare the scalablity of parameter learning techniques for different types of neural nets.  Since SPNs can be trained by SGD just like any other type of deep neural net, SPNs are as scalable as other types of network when the structure is fixed.
In fact, Poon and Domingos; UAI-2011 paper showed results where SPNs beat other types of deep neural nets on an image completion task.

Meta-review:
role: PC
data: comment: The authors present a framework for online structure learning of sum-product network. They overcome challenges such as being able to learn a valid sum product network and to have an online learning mechanism. Based on the extensive discussions presented by the reviewers, our recommendation is to accept this paper for a workshop.

Decision: Invite to Workshop Track