Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a novel method to regularize very large probabilistic circuits, mainly large sum-product networks (SPNs). The idea is to train a small external neural network to generate the parameters (mixture weights) of a large SPN. The neural network itself is defined by a conservative number of parameters. Given the structure of an SPN, the parameter generation process first learns low dimensional embeddings that are representative of groups of parameters (called sectors) in the SPN. These embeddings are then fed into the external neural network which outputs the actual parameters for the SPN. Both the embeddings and the external neural network can be jointly trained using gradient descent. The neural network imposes soft weight-sharing over the SPN parameters which results in limiting the degrees of freedom of the model, therefore reducing potential overfitting. On a variety of benchmark datasets, both discrete and continuous, the authors empirically evaluated the proposed technique to regularize large SPNs and found that their method outperformed the well-known weight decay regularizer in terms of accuracy (log-likelihood scores) with significantly fewer parameters.   

main_review: This paper makes an interesting novel contribution. The main goal is to reduce overfitting of large SPNs by limiting their degrees of freedom without limiting their expressivity. Learning smaller SPNs can reduce degrees of freedom at the cost of their expressivity. 
Here the authors propose an alternative method where the degrees of freedom of the model is indirectly controlled by a small-scale external neural network which offers to impose a soft weight-sharing strategy over the parameters. This soft weight-sharing does not necessarily reduce the total number of parameters (clusters) of the model but restricts their structure and thus restrictis their degrees of freedom. The proposed technique is not only an effective regularizer but also an efficient means to compactly represent SPNs. One can easily opt to store only the neural network parameters and the embeddings which together have much fewer parameters than the underlying SPN itself. As the authors described this has the added benefit of reducing the memory footprint of SPNs during their evaluation as well. 
 
The experiments seem quite comprehensive with a few exceptions. The authors tested their method on both discrete and continuous datasets and on two different structures (RAT-SPNs and EiNets). The results are promising. In majority of the cases the method reduced overfitting compared to the weight-decay strategy. The paper is written clearly and with quality. 

Questions/Comments to authors:
1) An important piece of study that is missing is the effect of the length of 'h'. Two values were used in the experiments (5 and 8).   
2) The values in tables 1 and 2 present average test log-likelihood?
3) The plot in figure 5 was quite informative. It will be useful if the scores for other datasets are also presented. 

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper presents HyperSPN, a way to reguralize the learning of Sum-Product networks (SPNs).
main_review: One advantage of the proposed approach is its memory efficiency.

The main problem of the paper regards the clarity of the proposed approach. 
- It is not clear how the embeddings are defined. The relation between the embeddings and the SPN is not defined. 
- The structure of the neural network used to learn the parameters is not defined
- The loss used to optimize the parameters is not reported
- How the embeddings are used by the neural network is not discussed

The experimental results prove an improvement of the proposed method when compared to weight decay. However, state of the art results on Twenty Datasets are not reported for a comparison.

Concluding, the idea seems to be interesting but the paper does not prove its validity. The approach should be better explained and the experiments should be extended.
limitations_and_societal_impact: The main limitation regrds the clarity of the proposed method
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work proposes a deep learning architecture, called HyperSPN, with built-in regularization. The architecture comprises two components: a probabilistic circuit (PC) and a neural network (NN). The main idea here is that the NN generates the parameters of the PC. The NN structure provides the regularization technique: parts of the NN embeddings are shared when generating the PC parameters.
The paper's theoretical contribution is the architecture involving the PC and NN for efficient training and inference tasks. Practical contributions include experiments in three different problem domains.
main_review: The originality of this work is not clear from the manuscript. The idea of conditioning a PC with weights from a NN has been explored in the PC literature. Some examples are [1] and, more recently, [2]. This paper can benefit from including a proper discussion on how the proposed method differs from those published. It would be interesting to discuss also how it compares to not recent but similar techniques, such as in [3].
The idea of regularizing the PC by sharing some of the NN architecture is smart and with promising results.

The paper's theoretical contributions are sound, to the best of my knowledge. There are only two derivations, namely Proposition 1 and Corollary 1, and both are well presented.

The manuscript is well written, and it has a good presentation. The architecture's description is supported by informative Figures 1, 2, and 3, and the experiments are also well described. This description is essential for the work's reproducibility.

The motivation of the manuscript is significant for the research community. Large models are prominent now in the literature and, thus, efficient regularization techniques are welcome. However, the manuscript should clarify this work's main contributions and how the work adds to previous similar papers. This clarification will strengthen this work's significance.
Another suggestion for improving significance is to show results on specific tasks. Improvements on log-likelihood, at times, do not reflect on improvements of particular tasks, such as classification or sampling. It would be interesting to see results on such tasks, besides the log-likelihood numbers.

Reference
* [1] "Conditional Sum-Product Networks: Imposing Structure on Deep Probabilistic Architectures." PDF: https://arxiv.org/abs/1905.08550
* [2] "RECOWNs: Probabilistic Circuits for Trustworthy Time Series Forecasting." PDF: https://arxiv.org/abs/2106.04148
* [3] "Hierarchical mixtures of experts and the EM algorithm." PDF: https://www.cs.toronto.edu/~hinton/absps/hme.pdf

limitations_and_societal_impact: The paper briefly discusses one limitation in the Conclusion section. Future works include combining HyperSPNs with techniques from memory-efficient back-propagation.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a novel method for regularizing large scale probabilistic circuits: The PC's parameters are partitioned into evenly sized sectors, and each sector is represented via a low dimensional embedding. A shared neural network is used to map the embeddings in to the PC's original parameter space. Experimental results show that this reduction in the number of trainable parameters yields improved generalization performance when compared to previous methods, mainly weight decay.
main_review: ### Originality / Significance
The proposed method is novel to my knowledge. It addresses a relevant problem, as the fast growth in the number of parameters is one of the main issues preventing RAT-SPNs and EiNets from scaling further.

The proposed method is intuitively plausible, and the theoretical analysis is correct.

### Evaluation
The proposed method is evaluated on a large variety of datasets, and the results are overall convincing.

Given that weight decay is the main point of comparison, it would be good run a hyperparameter search over its value, to verify that the chosen setting is sensible (and ideally optimal).

The robustness of the results in Tables 1 and 2 could also be improved by running the experiments multiple times and testing for significance.

For the SVHN experiment, it would be good to report the change in performance of the overall model, especially given the fact that the clusters are not uniform in size (i.e., given the reported data, it is not clear whether overall performance went up or down).

### Clarity
The paper is very well written and easy to follow.

### Summary
Overall, this paper makes a simple but useful contribution that is well presented, theoretically analyzed, and empirically evaluated.
I have therefore set my rating to accept, with the expectation that the authors will address the above comments on the empirical evaluation during the response phase.

limitations_and_societal_impact: The limitations and societal impact are adequately discussed.
review_title: null

Meta-review: Thank you for submitting your work to NeurIPS. The paper targets training large scale sum-product networks with a given structure. To this end, it pushes the idea of generating the mixture weights of
large PCs using a small-scale neural network. While there is a connection to Conditional SPNs as presented at PGM 2020, it actually covers the different aspect of learning SPNs and not conditional ones. It is very interesting to see that a shared neural network can be used during training to map the embeddings in to the PC's original parameter space. Scaling PCs (and not only conditional ones) is a really important question. Simple but highly effective and nice idea. 
Decision: Accept (Poster)