Title: Encoding and Decoding Representations with Sum- and Max-Product Networks
Authors: Antonio Vergari
Robert Peharz
Nicola Di Mauro
Floriana Esposito
Venue: ICLR 2017

Abstract: Sum-Product networks (SPNs) are expressive deep architectures for representing probability distributions, yet allowing exact and efficient inference. SPNs have been successfully applied in several domains, however always as black-box distribution estimators. In this paper, we argue that due to their recursive definition, SPNs can also be naturally employed as hierarchical feature extractors and thus for unsupervised representation learning. Moreover, when converted into Max-Product Networks (MPNs), it is possible to decode such representations back into the original input space. In this way, MPNs can be interpreted as a kind of generative autoencoder, even if they were never trained to reconstruct the input data. We show how these learned representations, if visualized, indeed correspond to "meaningful parts" of the training data. They also yield a large improvement when used in structured prediction tasks. As shown in extensive experiments, SPN and MPN encoding and decoding schemes prove very competitive  against the ones employing RBMs and other stacked autoencoder architectures.

TL;DR: Sum-Product Networks can be effectively employed for unsupervised representation learning, when turned into Max-Product Networks, they can also be used as encoder-decoders

Review discussion:
It would be helpful if the authors would explicitly list would the novel and key contributions of this paper are. I appears that previous work from the authors and others is covering a large portion of the approach employed here.
What prevents the authors to apply their approach to more interesting datasets?
Dear reviewer, thank you for your questions.
> It would be helpful if the authors would explicitly list would the
> novel and key contributions of this paper are. I appears that previous
> work from the authors and others is covering a large portion of the approach employed here.
The novel and key contributions of this work are:
- interpreting SPNs/MPNs as encoders AND decoders for RL
- devising a decoding scheme exploiting MPE inference and coping with missing activations
- a robust experimental evaluation for structured output prediction
Furthermore, we are now collecting more empirical results for the efficacy/resiliency of the decoding scheme with missing-at-random activations, which we can add to the paper, if needed.
We recognize that we could have better highlighted the differences from our previous work, "Visualizing and Understanding Sum-Product Networks" which is not published and freely accessible at arxiv (https://arxiv.org/abs/1608.08266).
We will update the paper in the following days to reflect this comment.
When in Section 3 we introduce concepts presented in the arxiv paper, we properly cite them.
In addition to the novel contributions listed above, that do not appear at all on the arxiv version, we also want to point out that:
- the arxiv version only looks at embedding extraction for SPNs only (no decoding, no MPNs)
- this work has a stronger and more cohesive focus on RL
- now we provide a stronger experimental settings for embedding evaluation (MLC, embeddings for X,Y, both, ...)
We do not know about other relevant work adopting our perspective for SPNs, yet if you point us to those references we will relate and properly compare our work to theirs.
> What prevents the authors to apply their approach to more
> interesting datasets?
'Interestingness' for a dataset may be subjective.
Even if the employed datasets are not on par for dimensionality with big data, they are still solid real world standard benchmarks for MLC and from the PGM community.
For instance, subsets of the 10 datasets are employed in the experiments of [1, 2, 3, 4] .
By employing them we were able to perform an extensive experimentation by training and tuning many learners in different learning approaches, on a common ground (for instance MADE can deal with binary data only)  even with limited computational resources. We believe an all-round experimentation on 10 datasets than on 1-2 bigger ones to be more suited to validate our claims.
Nevertheless, if you have some particular dataset we can try to run our approach on it or tell why we can't at the moment.
Moreover, we are in the process of scaling SPN structure learning and inference to much larger datasets.
However, we argue that the time to learn and tune deeper autoencoder models can be a stronger bottleneck than learning structure+weights for SPNs, if one has no prior knowledge.
[1] Dembczyński et al. (2012) On label dependence and loss minimization in multi-label classification, Machine Learning Journal 2012, Volume 88 (http://link.springer.com/article/10.1007/s10994-012-5285-8)
[2] Antonucci et al. (2013) An Ensemble of Bayesian Networks for Multilabel Classification, Proc. IJCAI 2013 (23rd Int. Joint Conf. on Artificial Intelligence)
[3] Kong et al. (2013) Transductive Multilabel Learning via Label Set Propagation,  IEEE Transactions on Knowledge and Data Engineering 25, 2013
[4] Zhou et al. (2012) Multi-instance multi-label learning, Artificial Intelligence 176, 2012
Like the other reviewer, I'm having a lot of trouble understanding exactly what this paper is doing...There seems to be the following process:
1. Given a dataset, construct an SPN with some algorithm (it seems like this algorithm implicitly is trying to generatively model the data.)
2.
Then the authors propose a scheme for constructing an embedding by running the SPN (just taking node activations?) and then also how to take those activations and regenerate the input from them.
The authors apply this in some combination on X and Y and do a lot of experiments, the result of which is not very clear. Can you please distill the results into a reasonably sized table?
For example, why would you predict E_{Y}?  That's just some function of the model?
In general, there is a lot of jargon (e.g. acronyms, shorthand, notation) in this paper making it very difficult to parse. Please don't write Q1, Q2, etc.
Thanks for your comments, I will try to clarify our approach better in the following comments:
> Given a dataset, construct an SPN with some algorithm (it seems like this algorithm implicitly is trying to generatively model the data.)
Yes, an SPN models a probability distribution p(\mathbf{V}) over some random variables \mathbf{V}. Their structure+parameters can be learned jointly in a very simple fashion, please refer to [1] to see how the generative model is built without directly optimizing the likelihood. We employ a variant of [1], see Appendix C.
In our experiments we have that \mathbf{V} = (\mathbf{X}, \mathbf{Y}), therefore we can learn a generative SPN modeling the features \mathbf{X} or modeling the labels \mathbf{Y} independently one from the other.
> Then the authors propose a scheme for constructing an embedding by running the SPN (just taking node activations?) and then also how to take those activations and regenerate the input from them.
As stated in Section 3, we build an embedding for a data instance by taking the SPN node activations (encoding phase), possibly discarding some nodes, e.g.  the leaves. Since nodes output valid probabilities, each embedding component has a clear and interpretable meaning (Section 3).
We employ these embeddings as new features for a discriminative task, as it is a common practice for Representation Learning.  Since we want to predict \mathbf{Y} given \mathbf{X}, we can exploit our encoded embeddings E_{X} and E_{Y} instead of the original random variables in different setting combinations (Section 5, first paragraph). We follow the idea of Representation Learning for which these embeddings would provide a better feature space than the original ones.
The decoding scheme we propose is crucial to exploit embeddings for
the labels \mathbf{Y} in such tasks.
First we learn an SPN over Y, then we turn it into an MPN as stated in Section 2. We then build E_{Y} by collecting activations from the MPN in the same way as before.
When some predicted embeddings (see next comment) are available, we employ the decoding algorithm (Section 4 and Appendix A) to get the actual predicted labels.
We employ MPE inference when the provided embeddings to be decoded have missing component values.
We learn embeddings E_{X} and E_{Y} with SPNs/MPNs to demonstrate that one can cheaply turn its generatively trained models  into valuable tools to extract meaningful representations for discriminative structured output prediction tasks.
> why would you predict E_{Y}?
Instead of learning a classifier to predict \mathbf{Y}, we can learn a classifier to predict E_{Y} and then use our MPN over  \mathbf{Y} to decode these predictions into the actual labels.
Again, the idea is that predicting E_{Y} can be more effective since the label dependencies are better disentangled in the embedding space.
Building label embeddings (and being able to decode them) is a common practice in Representation Learning and Multi-Label Classification.
For instance see [2], in which labels are embedded (compressed) in a lower dimensional space, then a regressor is trained to predict the embeddings and the predictions are decoded (decompressed) again.
See also [4], Section 2.3 for other references for different learning scenarios for label embeddings in the literature.
Note that the values reported in the tables are always the performances to predict \mathbf{Y} and not  E_{Y}.
> and do a lot of experiments, the result of which is not very clear
> Please don't write Q1, Q2
In a nutshell, we want to evaluate the proposed encoding/decoding scheme for structured output prediction and compare it to similar schemes for traditional generative probabilistic and autoencoder models.
The higher the classification accuracy, the more useful the embeddings+encoding/decoding approaches are.
We provide results for the EXACT MATCH in Section 5, which is the equivalent of the accuracy score for Multi-Label Classification [3]. We also show other metrics in the appendix such as the hamming score and the jaccard score (also known as subset accuracy) as we believe that it is important to have a broader look on other label dependencies, see [3]. If you need, we can run again our experiments with other performance metrics.
> Can you please distill the results into a reasonably sized table?
Table 1 is structured as follows: compared models (rows) are grouped by the learning setting (divided by horizontal lines), depending on if we employ embeddings for the \mathbf{X}, \mathbf{Y} (requiring a decoding phase) or both. The last two groups employ a k-NN as a way to decode E_{Y} to actual labels (Section 5, last paragraph). The last column provides the average rankings for the methods for each setting, the lower the better. In a nutshell, SPN and MPN embeddings are comparably good or better than other models, for each setting. We can highlight the lower rank for a better visual inspection.
We report the results in such a table because we feel that each learning setting can have its own winners, and it is worthwhile to have panoramic view on all the settings across all dataset. We have not found a more compact way to present all the results.
Nevertheless, if you suggest to remove/aggregate some specific columns or rows, or ask for a particular visual comparison we can provide it.
> In general, there is a lot of jargon (e.g. acronyms, shorthand,
> notation) in this paper making it very difficult to parse
We recognize that many acronyms and shorthand notation are used across the pages, mainly due to space constraints. In general, when we choose them we opted for the most commonly  employed one in the literature and only after having introduced the extended form.
However,if you point out the most difficult to parse parts, we will rewrite and/or explain better them to be easier to be read for the people not in the field.
[1] Gens et al. (2013) "Learning the Structure of Sum-Product Networks" ICML 2013
[2] Bathia et al. (2015) "Sparse Local Embeddings for Extreme Multi-label Classification" NIPS 2015
[3] Dembczyński et al. (2012) On label dependence and loss minimization in multi-label classification, Machine Learning Journal 2012, Volume 88
[4] Akata et al. (2013) Label-Embedding for Attribute-Based Classification, CVPR 2013
We updated the paper to take into account the reviewers' comments.
In particular, the following additions/modifications were made:
Section 1 Introduction:
- clarified the previous work cited from the arxiv paper, as suggested by AnonReviewer3
Section 3:
- added a citation to the arxiv paper
Section 4:
- better explained the decoding scheme (AnonReviewer1)
- explained the label embedding use (AnonReviewer1)
Section 5:
- divided the experiment tasks (AnonReviewer1)
- better highlighted the rank information in Table 1 (AnonReviewer1)
- added experiments to evaluate the resilience of the decoding scheme to missing at random embedding components
Appendix:
- added references to the datasets (AnonReviewer3)
- added figures for the new experiments
Please let us know if further modifications are needed.
The authors propose and evaluate using SPN's to generate embeddings of input and output variables, and using MPN to decode output embeddings to output variables. The advantage of predicting label embeddings is to decouple dependencies in the predicted space. The authors show experimentally that using SPN based embeddings is better than those produced by RBM's.
This paper is fairly dense and a bit hard to read. After the discussion, the main contributions of the authors are:
1. They propose the scheme of learning SPN's over Y and then using MPN's to decode the output, or just SPNs to embed X.
2. They propose how to decode MPN's with partial data.
3. They perform some analysis of when their scheme will lead to perfect encoding/decodings.
4. They run many, many experiments comparing various ways of using their proposed method to make predictions on multi-label classification datasets.
My main concerns with this paper are as follows:
- The point of this paper is about using generative models for representation learning. In their experiments, the main task is discriminative; e.g. predict multiple Y from X. The only discriminative baseline is a L2 regularized logistic regression, which does not have any structure on the output; it'd be nice to see how a discriminative structured prediction method would do, such as CRF or belief propagation.
- The many experiments suggest that their encoder/decoder scheme is working better than the alternatives; can you please give more details on the relative computation complexity of each method?
- One thing I'm still having trouble understanding is *why* this method works better than MADE and the other alternatives. Is it learning a better model of the distribution of Y? Is it better at separating out correlations in the output into individual nodes?  Does it have larger representations?
- I think the experiments are overkill and if anything, they way they are presented detract from the paper. There's already far too many numbers and graphs presented to be easy to understand.  If I have to dig through hundreds of numbers to figure out if your claim is correct, the paper is not clear enough. And, I said this before in my comments, please do not refer to Q1, Q2, etc. -- these shortcuts let you make the paper more dense with fewer words but at the cost of readability.
I *think* I convinced myself that your method works...I would love to see a table that shows, for each condition: (A) a baseline X->Y, (B) one *average* result across datasets for your method, and (C) one *average* result from a reasonable best competitor method. Please show for both the exact match and hamming losses, as that will demonstrate the gap between independent linear prediction and structured prediction. That would still be plenty of numbers but would make it much easier for the reader to verify your claims and you can put everything else in the Appendix.  E.g. something like:
Input | Predicted Output | Decoder | Hamming | Exact Match
X | P(Y) | CRF | xx.xx | xx.xx   (this is your baseline)
SPN E_X | P(Y) | n/a | xx.xx | xx.xx
X | SPN E_Y | MPN | xx.xx | xx.xx  (given X, predict E_Y, then decode it with an MPN)
Does a presentation like that make sense? It's just really hard and time-consuming for me as a reviewer to verify your results, the way you've laid them out currently.
Dear reviewer, thanks for your time and suggestions.
We will try to answer your questions more in detail, hoping to improve the paper towards a full acceptance.
> The point of this paper is about using generative models for representation learning. In their experiments, the main task is discriminative; e.g. predict multiple Y from X. The only discriminative baseline is a L2 regularized logistic regression, which does not have any structure on the output; it'd be nice to see how a discriminative structured prediction method would do, such as CRF or belief propagation.
It is true that the embeddings from our models (SPNs/MPNs, RBMs and MADEs) come from generative models, in an unsupervised way.
However, once these embeddings are extracted for the X/Y, a supervised linear model is trained on them (be it either a logistic or a ridge regressor). This is the main reason why we compare all our results against the 'plain' L2 logistic regressor baseline: to state how useful are the new embeddings compared to using none (the original X and Y), i.e. how decoupled and now linearly separable are the new representations learned in the previous unsupervised stage.
If needed, we can look into 'fully'-supervised non-linear discriminate models to provide additional comparisons.  One way to cope with the structure in the Y would be to look at Classifier Chains [1], we will also investigate CRFs as you suggest.
All in all, if they directly optimize a loss to predict the Y, we expect them to perform better.
> The many experiments suggest that their encoder/decoder scheme is working better than the alternatives; can you please give more details on the relative computation complexity of each method?
To extract an embedding from an SPN/MPN we simply evaluate them bottom-up, thus the complexity is linear in the size of the network, as it is for the other neural models.
Therefore, the main computational cost is learning the structure of the network with LearnSPN-b (see Appendix C). In that iterative scheme, the cost of clustering features (insert a product node) is quadratic in the number of the features, while clustering instances (insert a sum node) can be quadratic in the number of instances [2].
However, no more learning is required for SPNs/MPNs: since  both structure and weights are already learned.  We argue that such a learning step is 'cheaper' than performing intensive tuning for all the hyperparameters needed for specifying the structure of a neural model and learning its weight. E.g number of layers, hidden units, activation type, learning rate scheduling algorithm and coefficient, skipping connections and so on for MADEs, see the Appendix E.1.3.
> - One thing I'm still having trouble understanding is *why* this method works better than MADE and the other alternatives. Is it learning a better model of the distribution of Y?
If we take the log-likelihood as the criterion to measure how good a distribution is fit, the answer is no.
For instance MADE models have better log-likelihoods than SPNs in our experiments, since they directly optimize this loss while training, however their embeddings are less discriminative features.
We can report in additional tables the log-likelihoods achieved on the test sets by MADEs and SPNs. From this perspective RBMs are not directly comparable since their log-likelihood is intractable and we optimize the pseudo-log-likelihood instead. MANIAC, on the other hand, is not probabilitstic at all.
> Is it better at separating out correlations in the output into individual nodes?
We say yes, as we argue in the Introduction and Section 3 and 4. Our point is that SPN embeddings are inherently (probabilistic) *part-based* features that are extracted while performing that sort of hierachical co-clustering by Learn-SPN. From this point of view, we expect them to better capture local variations.
>  Does it have larger representations?
We do not believe it to be the cause, either. To demonstrate this, we accurately selected our model capacities in the experiments. Since LearnSPN-b adaptively grows a network, we first measured the embedding size from the learned SPNs and therefore trained RBMs/MADEs to have similar or bigger size.
Hence the need for models with different  hidden unit sizes.
The sizes of SPN embeddings are to be found in Appendix D.  The shortest and longest SPN embeddings on the X are ~21 and ~3530 respectively (Table 3, inner column), while on the Y we have ~25 and ~500 (Table 4, edges column).
>   I said this  before in my comments, please do not refer to Q1, Q2, etc. these shortcuts let you make the paper more dense with fewer words but at the cost of readability.
We will rewrite the experiment section to make it more clear with no shorthand notations for questions.
> I would love to see a table that shows, for each condition: (A) a baseline X->Y, (B) one *average* result across datasets for your method, and (C) one *average* result from a reasonable best competitor method. Please show for both the exact match and hamming losses, as that will demonstrate the gap between independent linear prediction and structured prediction. That would still be plenty of numbers but would make it much easier for the reader to verify your claims and you can put everything else in the Appendix.
We can refactor the table as suggested, adding a little of redundancy to the columns.
Concerning (A), at the moment we can provide the logistic regression baseline (LR, X->Y) since it actually is the most natural method to compare against, as stated above.
Concerning (B), the average rank we provided as the last column in Tables 1,7,8 is already an aggregate result that shows the best method across all datasets.
Nevertheless, we can average the result across all datasets as suggested. This is how the result table will look like: http://oi68.tinypic.com/2u786cp.jpg
Alternatively, to stress the improvement over the LR baseline we can report the average relative improvement of each score as follows:
avg_imp(j, s) = \sum_{d\in datasets} (s_{j}(d) - (s_{LR}(d))) * 100 / (s_{LR}(d))
where s(.) is the score function, j a model and s_{LR}(d) the score of the logistic regression baseline on the dataset d. The result table would look like this: http://oi67.tinypic.com/ve4osl.jpg
Concerning (C), we could aggregate rows across same methods (eg: RBM-500/1000/5000) by taking the max value, however losing the information that determining the embedding size does not come for free for all models but SPN/MPN. Here is how the table would look like: http://oi65.tinypic.com/301mx5e.jpg
What do you think? After this discussion we will insert the best table layour in a new revision.
[1] Read et al. (2011) "Classifier chains for multi-label classification." Machine learning 85.3
[2] Gens et al. (2013) "Learning the Structure of Sum-Product Networks" ICML 2013
Dear reviewer,
we updated the paper according to your suggestions and requests.
In particular, we added a comparison to max-margin CRFs as fully-supervised discriminative models; we presented the results in Table 1 following the layout you suggested and aggregated them across all datasets;
finally, we improved the overall experiment section incorporating the discussed aspects.
Moreover, we are further clarifying our work in the other reviewers' comment discussions.
Please let us know if further aspects need discussion.
The paper's aim is - as argued in the paper and the responses to other reviewers comments - that SPN and MPN can be interpreted as encoders and decoders of RL. Well - this is an interesting perspective and could be (potentially) worth a paper.
However
- the current draft is far from being convincing in that respect - and I am talking about the updated / improved version as of now. The paper does not require minor revisions to make this point apparent - but a significant and major rewrite which seems beyond what the authors have done so far.
- the experiments are (as also pointed out by other reviewers) rather unstructured and difficult to see much of an insight. I should probably also list (as the other other reviewers) flaws and issues with the experiments - but given the detailed comments by the other reviewers there seems to be little additional value in doing so
So in essence the paper simply does not deliver at this point on its promise (as far as I am concerned) and in that sense I suggest a very clear reject for this conference.
As for the dataset employed: MNIST should be considered a toy-dataset for pretty much all purposes these days - and in that sense the dataset choice is not helping me (and many other people that you might want to convince) to safe this paper.
Dear reviewer,
thanks for your comments up to now and for your appreciation of our idea to exploit SPNs/MPNs for RL.
> the current draft is far from being convincing in that respect - and I am talking about the updated / improved version as of now. The paper does not require minor revisions to make this point apparent - but a significant and major rewrite which seems beyond what the authors have done so far.
We acknowledge that the paper can be improved but we would appreciate somewhat more detailed comments. We are always eager to revise parts of the paper or add experiments.
We believe that this is a key advantage of conferences such as ICLR and the open review process. A revision of this work could surely benefit from constructive feedback.
> the experiments are (as also pointed out by other reviewers) rather unstructured and difficult to see much of an insight. I should probably also list (as the other other reviewers) flaws and issues with the experiments - but given the detailed comments by the other reviewers there seems to be little additional value in doing so
Please note that there is only one other reviewer, whose score is "Marginally above acceptance threshold".
While we accept your opinion concerning our experiments, we indeed would appreciate an objective list of flaws in order to substantiate your rating.
Please also note the proposed revision of our experiment section (see reply to Reviewer 1).
> As for the dataset employed: MNIST should be considered a toy-dataset for pretty much all purposes these days - and in that sense the dataset choice is not helping me (and many other people that you might want to convince) to safe this paper.
Please note that we employed MNIST only to show one aspect of the whole work: the visualizations of the part-based learned filters, for which an image dataset was needed.
However, we conducted our experiments on 10 more standard benchmark datasets for Multilabel-classification, since our major aim is to test embeddings for discriminative structured output prediction. Please see our answer to your pre-review question on this matter. Again, we are open for suggestions concerning additional datasets to be included in our experiments.
As a side note on learning the structure of an SPN on much larger (image) datasets, please consider that up to now, in the SPN literature, only (image) datasets on par with MNIST have been considered. Scaling structure learning would be worth a publication by itself.
All in all, we believe it is time to start researching tractable generative models for RL, even if we have to cope with all the (computational, theoretical, etc) limitations for these architectures that have been overcome for other now-used-everyday deep learning models.
Dear reviewer,
since your last comments on our paper we clarified our claims a lot by answering to the reviewers
questions.
Moreover, we changed the paper accordingly, trying to address your comments.
In particular, we extended and refactored the experimental section.
We would like to know whether even this new version continues to be flawed in your opinion.
This paper tries to solve the problem of interpretable representations with focus on Sum Product Networks.  The authors argue that SPNs are a powerful linear models that are able to learn parts and their combinations, however, their representations havent been fully exploited by generating embeddings.
Pros:
-The idea is interesting and interpretable models/representations is an important topic.
-Generating embeddings to interpret SPNs is a novel idea.
-The experiments are interesting but could be extended.
Cons:
-The author's contribution isn't fully clear and there are multiple claims that need support.
For example, SPNs are indeed interpretable as is, since the bottom-up propagation of information from the visible inputs could be visualized at every stage, and the top-down parse could be also visualized as it has been done before (Amer & Todorovic, 2015).
Another example, Proposition one claims that MPNs are perfect encoder decoders since the max nodes always have one max value, however, what if it was uniformally distributed node, or there are two equal values? Did the authors run into such cases? Did they address all edge cases?
-A good comparison could have been against Generative Adversarial Networks (GANs), Generative Stochastic Networks (GSNs) and Variational Autoencoders too since they are the state-of-the-art generative models, rather than comparing with RBMs and Nade.
I would suggest that the authors take sometime to evaluate their approach against the suggested methods, and make sure to clarify their contributions and eliminate over claiming statements. I agree with the other comments raised by Anon-Reviewer1.
Dear reviewer, thank you for your comments and for showing interest in our approach.
In the following we clarify our choices and claims.
> SPNs are indeed interpretable as is, since the bottom-up propagation
> of information from the visible inputs could be visualized at every
> stage, and the top-down parse could be also visualized as it has
> been done before (Amer & Todorovic, 2015)
Indeed, SPNs have a clear, full probabilistic semantics but up to now they have been employed as classical probabilistic models in which interpretability is tackled only by looking at the results of single probabilistic queries, be them full evidence or MPE inference.
To substantiate how our aim is different, consider the the work in (Amer & Todorovic, 2015). By applying an approximate MPE inference routine they i) infer the most probable activity from a video sequence and ii) map that activity to a foreground region in the video. They are weakly/fully supervised to infer the values of the indicator variables they set in a fixed-a-priori grid to locate these foreground/background regions.
Instead, we train our SPNs in an unsupervised fashion, extracting structured embeddings from inner nodes jointly and employing them for predictive tasks on unseen variables.
Moreover, by effectively visualizing the features extracted by each node, we provide an evidence that each node acts a meaningful part-based filter.
We already investigated visualizing how inference propagates in SPNs at different stages in our unpublished arxiv paper (Vergari et al. 2016). If needed, we could add to this work more visualizations from it.
> Proposition one claims that MPNs are perfect encoder decoders since the max nodes always have one max value, however, what if it was uniformally distributed node, or there are two equal values?
Please note that Proposition one claims that MPNs are perfect encoder-decoders *only if* there are no ties among child distributions. This is always true for deterministic (selective) MPNs (Proposition 2) but may not be true in practice for non-deterministic MPNs learned from data.
> Did the authors run into such cases? Did they address all edge cases?
Indeed, we investigated this issue empirically for our networks in Section 5.1 when employing full embeddings.  Please refer to tables 5 and 6 to see how close to perfect encoder-decoders our MPNs were on the X and Y.
> -A good comparison could have been against Generative Adversarial Networks (GANs), Generative Stochastic Networks (GSNs) and Variational Autoencoders too since they are the state-of-the-art generative models, rather than comparing with RBMs and Nade
To support our claim that probabilistic representations from tractable models like SPNs/MPNs are indeed meaningful, we devised our experimentation to provide fair competitors.
While GANs, VAEs and even GSNs are state-of-the art models *for generating* high quality samples, their likelihood is intractable (even implicit for GANs) and exact inference is unfeasible with them. As a consequence, computing probabilistic features for unsupervised learning comparable to those extracted from SPNs in Section 3 would be not immediate.
On the other hand, MADEs admit tractable exact full evidence inference, and they provide a way to decode the learned representations. This makes them a very natural competitor for our SPNs/MPNs encoding and decoding schemes. RBMs, while having an intractable joint likelihood, admit tractable conditionals of the hidden variables given the evidence and such probabilistic features have proved to be very competitive in the literature.
We will discuss these differences in more detail in a new revision.
Dear reviewers, we have updated the paper following your suggestions.
The new revision contains the following updates:
- Overall refactoring of the experiment section, improving the presentation (AnonReviewer1, AnonReviewer2)
- Removed references to Q1, Q2, etc (AnonReviewer1)
- Added CRFs as a fully supervised and discriminative method to the experiments (AnonReviewer1)
- Extended the discussion on structured output prediction into Section 5.2.1 discussing 'why' the performance improvements (AnonReviewer1)
- Restructured Table 1 to hold aggregated relative improvements w.r.t LR for all datasets (AnonReviewer1)
- Refactored the experimental setting explanation, better motivating the choice of the competitors (AnonReviewer2)

Meta-review:
role: AC
data: comment: Dear authors, the reviewers brought up an interesting point in their reviews.  They would like to understand the choice of benchmarks and baselines.  Specifically, the comparisons in this paper focus on MADE, NADE and MANIAC.  MANIAC seems like a curious choice, since as far as I know, it is not well known in the community.  NADE and MADE seem like good state-of-the-art comparisons.  However, these models are designed to estimate the probability of the data given the model, and as such report the log-likelihood of the data in their experiments.  The experiments in this paper seem to focus on multi-label classification accuracy.  This difference would suggest that other baselines, which are designed for that purpose, would be more appropriate.  Could you clarify why multi-label classification accuracy was chosen as the metric under which to compare this model to these specific baselines?  Since a major claim is that the model is probabilistic, wouldn't it at least be much more natural to report log-likelihood of the correct classification rather than zero-one error?

Another point which makes understanding the experiments difficult is the introduction of new datasets.  As far as I can tell, neither NADE or MADE run experiments on the datasets used in these papers.  However, those models' results are reported on other datasets.  Reporting results on the same problems as the state-of-the-art models would make the experiments much more compelling.  Why was this particular data set chosen?  Why was e.g. the "Cal" data set included (as far as I can tell every method gets 0 errors).
role: Author
data: value: comment: Dear reviewer, thanks for your questions.
We hope to clarify our experimental choices.

We recognize that our experiment setting might appear *unusual* if we were to compare different probabilistic models w.r.t. their capability of modeling a probability distribution. However, here our aim is different: to exploit generative probabilistic models as unsupervised feature extractors for Representation Learning.

Specifically, given a dataset X, we train an unsupervised model f on X, which we employ to extract a new representation (embedding) E_X = f(X). We re-use such representation, later, in *other tasks*, such as structured output prediction in our case. The unsupervised model f could correspond to any generative model for which probabilistic features are tractable, e.g. SPNs/MPNs, RBMs, MADEs, etc as employed in the experiments. 

Apart from using generative models, this is a two-staged approach that is common to several others in RL when features are extracted from a model then feeded into classifiers such as SVM, LR, etc (e.g. [1] using auto-encoders and RBMs, [2] extracting image features with a CNN from one dataset and using them for classification but on another dataset, or exploiting them for different predictive tasks like segmentation, salience detection, etc [3, 4]).

We then employ the predictive performance of a simple linear model trained on these new embeddings instead of the raw  ones as a proxy to measure their meaningfulness.

Our main claim is that probabilistic generative models as SPNs/MPNs can be used to extract and decode features after they have been learned unsupervisedly. The capabilities of SPNs as density estimators in terms of the likelihood have been  already assessed in previous literature.


> Could you clarify why multi-label classification accuracy was chosen
> as the metric under which to compare this model to these specific
> baselines?  Since a major claim is that the model is probabilistic,
> wouldn't it at least be much more natural to report log-likelihood
> of the correct classification rather than zero-one error?

As already state above, the aim is to assess whether the SPNs learned representations could be beneficial to build classification models. 
   
Hence, we chose MLC as the predictive task in our experiment since it is inherently harder than binary and multi-class classification. Moreover and more interestingly, MLC allows us to build embeddings  not only for the input but also for the label space, enabling several settings in which to evaluate our approaches.

Due to the MLC task, we employ metrics like EXACT MATCH, HAMMING and JACCARD (aka subset accuracy) to evaluate the linear models' predictive performances, and hence the usefulness of the learned representations employed.

While the higher (log-)likelihood of the probabilistic models employed would measure their ability to answer probabilistic queries with higher fidelity, i.e. they ability to better model the probability distribution, it would not give us information about the meaningfulness of the learned representations as features to be employed in other predictive tasks. 


> NADE and MADE seem like good state-of-the-art comparisons.  However,
> these models are designed to estimate the probability of the data
> given the model, and as such report the log-likelihood of the data
> in their experiments

MADE provides a natural competitor for our approaches, because it is  generative model (i) that is also an autoencoder (ii). In this way, we can both evaluate the meaningfulness of extracting features form a state-of-the-art probabilistic model and we can decode them back when we are dealing with label embeddings in the proposed approaches.


> MANIAC seems like a curious choice, since as far as I know, it is
> not well known in the community.

We added MANIAC to the comparison even if it is not a probabilistic model and not well known in the ICRL community because it is a deep autoencoder (i), it has been tailored for MLC (ii) and it operates in the very same two stage as we do (iii): first label embeddings are learned with a deep autoencoder, then a linear model is trained to predict them, finally a these predictions are decoded back.
 
 
> This difference would suggest that other baselines, which are designed for that purpose, would be more appropriate

To measure the meaningfulness of extracted representations for all competitors, we compare the performance of a linear model trained on them (in several settings) against that of a linear model on the original features. The better the new representation disentangle the data dependencies, the higher the improvement a linear model could score if trained on them.

As AnonReviewer1 asked, we also added as a baseline the performance on our datasets of a CRF as a fully-supervised and discriminative model, hence directly optimizing the predictive score from the start, differently from our two-stage approach.

If other comparable methods are suggested, we can try to add them or tell why we cannot.

> Why was this particular data set chosen?  

Since we focused on MLC as the predictive task for our learned representation evaluation, we adopted standard MLC benchmark datasets. The Appendix and our answers to AnonReviewer2 comment detail the references about these datasets.

> As far as I can tell, neither NADE or MADE run experiments on the datasets used in these papers.  However, those models' results are reported on other datasets.  Reporting results on the same problems as the state-of-the-art models would make the experiments much more compelling

In their original paper MADEs have been evaluated on standard binary datasets for density estimation, i.e. they have been evaluated only for their capacity of modeling p(X) (by reporting the likelihoods only). Except for the binarized version of MNIST, these datasets do not have label information.

If we wanted to use those same datasets for evaluating our models as feature extractors for MLC, we would have create the labels by ourselves in some way (e.g. by selecting a subset of the original X features to consider them as labels Y). This might have been more arbitrary than employing standard datasets for MLC.

> Why was e.g. the "Cal" data set included (as far as I can tell every method gets 0 errors).
Cal, also known as Cal500, is usually known to be a hard dataset for MLC w.r.t the EXACT MATCH score. This does not happen for the JACCARD and HAMMING scores (reported in the Appendix) since they are more forgiving scores that do not require all labels to be predicted exactly. To the best of our knowledge, no model has scored more than 0.0 EXACT MATCH on Cal.

We updated our work with a new revision, please tell us if in this way we answered your questions.


[1] Coates et al. (2011)  "An analysis of single-layer networks in unsupervised feature learning" JMLR 
[2] Simonyan et al. (2015) "Very Deep Convolutional Networks for Large-Scale Image Recognition" ICLR 2015
[3] Kümmerer et al. (2015) "Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet" ICLR workshop 2015
[4] Girshick et al. (2014) "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation" CVPR 2014

role: AC
data: value: comment: Dear author, thanks for your clarification.  This certainly helps to understand the paper a bit better.

The concern w.r.t. baselines and benchmarks is the following.  NADE and MADE were designed to be good models of p(X) as you said.  My understanding is that this is also the purpose of your model (i.e. a generative model of the data).  However, you don't compare under the metric which is (as is my understanding) the purpose for which your model (and those) is designed.  Instead, your comparison is made using a series of multi-label classification tasks.  The assumption there is that better (or more useful?) latent representations of the data would necessarily lead to better classification results (I'm not sure I agree with this entirely).   However, you don't need a generative model or p(X) to do this well.  The literature of autoencoders explores and evaluates on this task extensively (on MNIST, CIFAR-10, CIFAR-100, Imagenet, etc.).  So the question is, why don't you compare to e.g. families of autoencoders if creating an intermediate representation that is good for classification is your task?

role: Author
data: value: comment: Dear reviewer, thanks for the quick reply and for the opportunity of clarifying our claims further. We really appreciated it.

In summary, our research objective is to demonstrate that generatively trained SPNs/MPNs can also be exploited as encoders or encoder/decoders for useful features that, when used in discriminative tasks, yield comparable or better performance than other generative models used for the same purpose and other non-probabilistic autoencoder architectures as well.
We will try to further substantiate how we arranged our experiments to achieve this objective.


> NADE and MADE were designed to be good models of p(X) as you said.  My understanding is that this is also the purpose of your model (i.e. a generative model of the data).
> However, you don't compare under the metric which is (as is my understanding) the purpose for which your model (and those) is designed.

Indeed, this is correct. This is also the case for RBMs, which are trained to optimize the (pseudo-)likelihood of the data.
Even if they are optimized for such a generative and unsupervised task, the features they provide have been successfully employed in discriminative and supervised tasks. Consider, for instance, approaches like ours in which a linear classifier is trained on top of RBM features in [1][2][3]. Also the seminal works by Hinton [4] show how these features extracted from RBMs optimizing a certain loss can be successfully used to initialize deep architectures to be trained according to another loss.


> However, you don't need a generative model or p(X) to do this well.

Indeed, it is correct that in order to extract useful representation it *is not necessary* that the extractor shall be a generative model. However, in the literature *also* generative models have been proved to be solid feature extractors. Consider the case of RBMs as quoted above, for instance.

As we stated in the introduction, generative models can *potentially* be used for representation learning. As already mentioned, it is our aim to verify this aspect also for SPNs/MPNs.  The main advantage that, in such an approach, a generative model gives you is that you can also employ it for answering many probabilistic queries. With SPNs/MPNs the additional advantage one get is that many probabilistic queries (e.g. marginals, partition function, etc) are exact and tractable (differently from RBMs, NADEs, etc). If we verify that they can be used also as unsupervised feature extractor, one can learn them *once* and exploit them in several ways.

We recognize that MADEs have not been exploited as feature extractors in the literature, up to now. We believe that they can potentially be employed in such a way. Our experiments on embedding X confirmed this: MADEs proved to perform better than RBMs. Moreover, we chose them, as stated in the previous comment, because not only they are generative models, but also deep autoencoders (see below).

> The literature of autoencoders explores and evaluates on this task
> extensively (on MNIST, CIFAR-10, CIFAR-100, Imagenet, etc.). 
> So the question is, why don't you compare to e.g. families of autoencoders if creating an intermediate representation that is good for classification is your task?

Indeed it is true that autoencoders are a needed model to compare against in our experiments.

To clarify our position, our aim is not to produce state-of-the-art results for those MLC benchmarks, but to assess the ability of a new generative model to provide meaningful representations in a very 'cheap' fashion.
More specifically, other generative models as feature extracts are needed as a baseline *relative* to this newly introduced model. Non-probabilistic models are needed to set a baseline for all the probabilistic models we employ.

Concerning our choices, we did included two kinds of (deep) autoencoders: probabilistic autoencoders like MADEs, which we employed in all the settings, and deterministic autoencoders in the form of MANIAC. In addition to what we said about our choice of MANIAC in the previous comment, we highlight that, despite its name, its architecture is that of classic stacked autoencoders, but employed for label embeddings.

We agree with you about adding other kinds of autoencoders into the experiments, to give a better *global perspective* of the results. If you suggest some particular family of autoencoders to experiment with, we will try to perform our experiments with them, in this remaining time.

Concerning the datasets employed, if needed, we can also already report the classification accuracy on MNIST, Caltech-101, OCR letters, Convex and Rectangles [5] for SPN, MADE and RBM embeddings. These are binary image datasets that are better known in the literature for models that accept only binary RVs, like MADEs. We reported these results on our arxiv paper, where we focused only on a simpler predictive task as multi-class classification.


[1] Ranzato et al. (2008) "Sparse Feature Learning for Deep Belief Networks" NIPS 2008
[2] Marlin et al. (2010) "Inductive Principles for Restricted Boltzmann Machine Learning" AISTATS, 2010
[3] Ranzato et al. (2010) "Modeling pixel means and covariances using factorized third-order Boltzmann machines" CVPR 2010
[4] Hinton et al. (2006) "Reducing the dimensionality of data with neural networks" Science 2006
[5] Larochelle et al. (2007) "An Empirical Evaluation of Deep Architectures on Problems with Many Factors of Variation" ICML 2007

role: PC
data: comment: Dear authors, in general the reviewers found that the paper was interesting and has potential but needs additional work in the presentation and experiments. Unfortunately, even if all reviews had been a weak accept (i.e. all 6s) it would not have met the very competitive standard for this year.
 
 A general concern among the reviewers was the presentation of the research, the paper and the experiments. Too much of the text was dedicated to the explanation of concepts which should be considered to be general knowledge to the ICLR audience (for example the justification for and description of generative models). That text could be replaced with further analysis and justification.
 
 The choice of baseline comparisons and benchmarks did not seem appropriate given the presented model and text. Specifically, it is difficult to determine how good of a generative model it is if the authors don't compare it to other generative models in terms of data likelihood under the model. Similarly, it's difficult to place it in the literature as a model for representation learning if it isn't compared to the state-of-the-art for RL on standard benchmarks.
 
 The clarifications of the authors and revisions to the manuscript are greatly appreciated. Hopefully this will help the authors to improve the manuscript and submit to another conference in the near future.

Decision: Reject