Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper tries to solve the problem of interpretable representations with focus on Sum Product Networks.  The authors argue that SPNs are a powerful linear models that are able to learn parts and their combinations, however, their representations havent been fully exploited by generating embeddings.

Pros:
-The idea is interesting and interpretable models/representations is an important topic.
-Generating embeddings to interpret SPNs is a novel idea.
-The experiments are interesting but could be extended.

Cons:
-The author's contribution isn't fully clear and there are multiple claims that need support. For example, SPNs are indeed interpretable as is, since the bottom-up propagation of information from the visible inputs could be visualized at every stage, and the top-down parse could be also visualized as it has been done before (Amer & Todorovic, 2015). Another example, Proposition one claims that MPNs are perfect encoder decoders since the max nodes always have one max value, however, what if it was uniformally distributed node, or there are two equal values? Did the authors run into such cases? Did they address all edge cases? 

-A good comparison could have been against Generative Adversarial Networks (GANs), Generative Stochastic Networks (GSNs) and Variational Autoencoders too since they are the state-of-the-art generative models, rather than comparing with RBMs and Nade.

I would suggest that the authors take sometime to evaluate their approach against the suggested methods, and make sure to clarify their contributions and eliminate over claiming statements. I agree with the other comments raised by Anon-Reviewer1.
review_title: Interesting work with limited contribution.

final_score: rating: 3: Clear rejection
review_content: 
The paper's aim is - as argued in the paper and the responses to other reviewers comments - that SPN and MPN can be interpreted as encoders and decoders of RL. Well - this is an interesting perspective and could be (potentially) worth a paper. 

However

- the current draft is far from being convincing in that respect - and I am talking about the updated / improved version as of now. The paper does not require minor revisions to make this point apparent - but a significant and major rewrite which seems beyond what the authors have done so far. 

- the experiments are (as also pointed out by other reviewers) rather unstructured and difficult to see much of an insight. I should probably also list (as the other other reviewers) flaws and issues with the experiments - but given the detailed comments by the other reviewers there seems to be little additional value in doing so

So in essence the paper simply does not deliver at this point on its promise (as far as I am concerned) and in that sense I suggest a very clear reject for this conference. 

As for the dataset employed: MNIST should be considered a toy-dataset for pretty much all purposes these days - and in that sense the dataset choice is not helping me (and many other people that you might want to convince) to safe this paper. 
review_title: unconvinced on multiple fronts

final_score: rating: null
review_content: Like the other reviewer, I'm having a lot of trouble understanding exactly what this paper is doing...There seems to be the following process:

1. Given a dataset, construct an SPN with some algorithm (it seems like this algorithm implicitly is trying to generatively model the data.)
2. Then the authors propose a scheme for constructing an embedding by running the SPN (just taking node activations?) and then also how to take those activations and regenerate the input from them.

The authors apply this in some combination on X and Y and do a lot of experiments, the result of which is not very clear. Can you please distill the results into a reasonably sized table? For example, why would you predict E_{Y}?  That's just some function of the model? 

In general, there is a lot of jargon (e.g. acronyms, shorthand, notation) in this paper making it very difficult to parse. Please don't write Q1, Q2, etc. 

review_title: Please clarify the story of the paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose and evaluate using SPN's to generate embeddings of input and output variables, and using MPN to decode output embeddings to output variables. The advantage of predicting label embeddings is to decouple dependencies in the predicted space. The authors show experimentally that using SPN based embeddings is better than those produced by RBM's.

This paper is fairly dense and a bit hard to read. After the discussion, the main contributions of the authors are:

1. They propose the scheme of learning SPN's over Y and then using MPN's to decode the output, or just SPNs to embed X.
2. They propose how to decode MPN's with partial data.
3. They perform some analysis of when their scheme will lead to perfect encoding/decodings.
4. They run many, many experiments comparing various ways of using their proposed method to make predictions on multi-label classification datasets.

My main concerns with this paper are as follows:

- The point of this paper is about using generative models for representation learning. In their experiments, the main task is discriminative; e.g. predict multiple Y from X. The only discriminative baseline is a L2 regularized logistic regression, which does not have any structure on the output; it'd be nice to see how a discriminative structured prediction method would do, such as CRF or belief propagation. 

- The many experiments suggest that their encoder/decoder scheme is working better than the alternatives; can you please give more details on the relative computation complexity of each method?

- One thing I'm still having trouble understanding is *why* this method works better than MADE and the other alternatives. Is it learning a better model of the distribution of Y? Is it better at separating out correlations in the output into individual nodes?  Does it have larger representations? 

- I think the experiments are overkill and if anything, they way they are presented detract from the paper. There's already far too many numbers and graphs presented to be easy to understand.  If I have to dig through hundreds of numbers to figure out if your claim is correct, the paper is not clear enough. And, I said this before in my comments, please do not refer to Q1, Q2, etc. -- these shortcuts let you make the paper more dense with fewer words but at the cost of readability.

I *think* I convinced myself that your method works...I would love to see a table that shows, for each condition: (A) a baseline X->Y, (B) one *average* result across datasets for your method, and (C) one *average* result from a reasonable best competitor method. Please show for both the exact match and hamming losses, as that will demonstrate the gap between independent linear prediction and structured prediction. That would still be plenty of numbers but would make it much easier for the reader to verify your claims and you can put everything else in the Appendix.  E.g. something like:

Input | Predicted Output | Decoder | Hamming | Exact Match
----
X | P(Y) | CRF | xx.xx | xx.xx   (this is your baseline)
SPN E_X | P(Y) | n/a | xx.xx | xx.xx 
X | SPN E_Y | MPN | xx.xx | xx.xx  (given X, predict E_Y, then decode it with an MPN)

Does a presentation like that make sense? It's just really hard and time-consuming for me as a reviewer to verify your results, the way you've laid them out currently.



review_title: Final review

final_score: rating: null
review_content: 
It would be helpful if the authors would explicitly list would the novel and key contributions of this paper are. I appears that previous work from the authors and others is covering a large portion of the approach employed here.

What prevents the authors to apply their approach to more interesting datasets?
review_title: What are the key contributions of this paper?

Meta-review: Dear authors, in general the reviewers found that the paper was interesting and has potential but needs additional work in the presentation and experiments. Unfortunately, even if all reviews had been a weak accept (i.e. all 6s) it would not have met the very competitive standard for this year.
 
 A general concern among the reviewers was the presentation of the research, the paper and the experiments. Too much of the text was dedicated to the explanation of concepts which should be considered to be general knowledge to the ICLR audience (for example the justification for and description of generative models). That text could be replaced with further analysis and justification.
 
 The choice of baseline comparisons and benchmarks did not seem appropriate given the presented model and text. Specifically, it is difficult to determine how good of a generative model it is if the authors don't compare it to other generative models in terms of data likelihood under the model. Similarly, it's difficult to place it in the literature as a model for representation learning if it isn't compared to the state-of-the-art for RL on standard benchmarks.
 
 The clarifications of the authors and revisions to the manuscript are greatly appreciated. Hopefully this will help the authors to improve the manuscript and submit to another conference in the near future.
Decision: Reject