Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper considers the problem of learning tractable probabilistic models (TPMs) and presents a gradient-based method for learning a TPM from local (possibly inconsistent) estimates. Specifically, a TPM is first learnt from the input dataset and is subsequently refined by updating its parameters using a set of local probability estimates defined over small subsets of variables (i.e., local marginal distributions). The parameters are updated using a gradient descent based approach with an objective function that ensures closeness to the local estimates as well as the learnt model. The empirical evaluation demonstrates that the proposed method produces tractable models that are superior to those learned solely from data.

strengths_and_weaknesses: The paper is fairly well written and organised. The quality of the presentation is overall very good and therefore the paper is relatively easy to follow. Most of the concepts are introduced and discussed if a fairly clear manner. The empirical evaluation is sound and demonstrates clearly the strength of the proposed method.

questions: 1) How realistic is the assumption that the marginal probabilistic estimates are available? Could you describe a real-world application where this information is readily available? The experimental setup in Section 5, especially the generation of the local estimates, looks a bit artificial.

2) I was wondering how previous methods like IPFP perform at least for the case when the local estimates are consistent.

3) How sensitive is the method to the learning rate? Did you try using an adaptive learning rate?

[Post Rebuttal] Thanks for your answers. They addressed my concerns.
limitations: see above
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper investigates a learning setting where training data is paired with noisy local estimates, focusing on Cutset Networks, a popular class of tractable probabilistic models (although the results seem easily extendable to other TPM classes).

Tractable models enable the efficient evaluation of the gradients required to solve this learning problem. The paper then proposes an iterative, gradient ascent approach. To further speed up convergence, instead of maximizing the original objective involving both the local estimates and training data, the paper proposes a moment-matching variant where the local estimates are considered as a post-processing step after training the CN on data using standard techniques. 

strengths_and_weaknesses: 
I found the paper generally well-written. I think it is quite accessible even for non-experts in tractable models while being sufficiently detailed in the description of the contribution. The empirical evaluation is also convincing.

---

I am not completely convinced that having access to these local estimates is "quite common in real-world applications". The work cited
at lines 38-39 is not specifically supporting the claim. Considering that prior work in this setting is quite limited (according to the Related Work section), it would be nice if the paper substantiated more this claim with real-world examples.

[Updated after rebuttal] The authors addressed my concerns in their response and included the changes in an updated version of the paper.

questions: 1- Can you further substantiate that local estimates can be easily obtained in many real-world applications?

2- Why not including a comparison of the proposed moment-matching approach with a variant where stochastic gradient ascent is used to optimize the objective in Eq. 5?

---

Minor comment:

"To avoid clutter, we do not report the standard deviation because it was fairly small over all runs."

- I would still report how (relatively) small it is or add the full table in the supplementary material.


limitations: I did not find any unaddressed limitations in the paper. I am looking forward to reading an answer to Q1 in order to better assess the impact of this work, possibly raising my 'Contribution' and overall rating.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an algorithm for learning a tractable probabilistic model (TPM) from a dataset consisting of (1) a (possibly small) complete dataset and (2) (possibly many) local marginal estimates.  The tractable probabilistic model is a cutset network (or mixtures of).  They propose a tractable gradient algorithm for learning such a model, and show empirically how their algorithm can significantly improve the quality of a learned model when local marginal estimates can be taken into account.

Cutset networks are tractable; given complete data their parameters can be learned in closed form, and otherwise their gradients can also be computed in polynomial time.  The learning algorithm is based on gradient descent, where the local marginal estimates are incorporated as a soft constraint in the objective function.

As the authors have mentioned, the learning problem (or variations of it) have been considered before in the probabilistic graphical models literature, but there are some fundamental difficulties with this learning task on Bayesian networks and Markov networks.  Namely, checking whether local marginal constraints are satisfied requires inference in the network, which is computationall hard for BNs/MNs but can be done in polytime on tractable models such as cutset networks.

To my knowledge, the exploitation of a tractable probablistic model (TPM) for this learning task is novel.  There are certainly practitioners who are interested in this learning task for their applications, and I believe this type of paper would help spread the practical use of TPMs.

strengths_and_weaknesses: The proposed problem (learning a probabilistic model from data + marginals) is relevant in practice, and the authors apply a more modern class of (tractable) statistical models that is able to overcome limitations of classical models (like Bayesian networks).  The experiments demonstrate the utility on the learning problem.  Because of the (computational) limitations inherent in Bayesian networks, one did not see this learning scenario as often in practice, but given the use of tractable models, it may see increased visibility.

questions: I am familiar with this setting, and the developments to me were clear enough to me with no big surprises.

limitations: no negative societal impact, and otherwise limitations and relations to related work were satisfactory.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes to incorporate so-called local estimates into the optimization of probabilistic models. Since such estimates are typically intractable, the paper focuses on using tractable probabilistic models in order to enforce them into the loss function. The main discussion centers on learning cutset networks with (possible inconsistent) local estimates. To do so the paper formulates this as a constrained optimization problem (constraints being the local estimates) and takes a Lagrangian relaxation approach which basically ends up looking like a weighted joint optimization problem. Finally, due to the properties of cutset networks where parameters map to conditional probabilities, the loss function variance can be reduced through moment-matching instead of SGD, leading to faster convergence. Experiments are run on the 20-Datasets benchmark, showing that incorporating local estimates leads to better learned models when evaluated on KL divergence against an oracle model (chosen to be a mixture of cutset networks).
strengths_and_weaknesses: My overall opinion of this paper is poor. The paper has three significant and pervasive shortcomings.

From start to finish, the assumption of local estimates felt impractical and not well-justified. Are there really many scenarios where we would have large access to local estimates? If so, why are there no experiments on real datasets with real local estimates? The current experiments are completely synthetic, in the sense that the paper learns an oracle model using the full dataset to synthetically generate the local estimates, and then synthetically hides partial dataset from the learner models. 

Why are the experiments only testing Q vs R, with no reasonable baselines? It is completely expected that R will do better than Q, since it is given strictly more information (essentially local statistics from the oracle model)? Where are the ablations using SGD, using other TPMs, or using any other method to incorporate local estimates? Altogether, I find it the experimental evaluation sorely lacking.

Why is there such emphasis on cutset networks? The paper tries to start off general, making statements that try to encompass all TPMs, but the entirety of the experiments end up being focused on cutset networks. Even the moment-matching technique is specific to cutset networks. General TPMs do not have the conditional distribution interpretation in their parameters -- would probably need to rely on SGD for optimization. Given this, there should absolutely have been studies ran on other families of TPMs. As it stands, the paper (and its title) should advertise only cutset networks.

- on this, why don't we use general probabilistic circuits? What are the drawbacks of PCs compared to cutset networks?



**Strengths**

- the formulation is reasonable, and the derivations look correct
- the writing is reasonably clear

**Weaknesses**

- poor justification of access to local estimates. At the very least there should be experiments on datasets with real local estimates, not synthetically generated.
- experiments section is weak and far from 'detailed'. Missing ablations (SGD instead of moment matching), other TPMs, choices of oracle model...
- unjustified choice of the heavy emphasis on cutset networks.

questions: .
limitations: .
review_title: null

Meta-review: metareview: This submission did not reach a full agreement among PC members. I will not repeat the arguments here as they can be read in the reviews (please see e.g. review WY8E, which according to the authors captures well their intention). The main open criticism is the lack of a real application where the type of assumption used in the paper is present (the other important one being the comparison with other approaches, which seems to have been justified by the authors to a good extent). The concern was originally about the existence of those applications, but later this was resolved (it remained as a concern that the real application is not shown in this work). I consider that this is small when weighted against the positive comments in all reviews.
confidence: Less certain
Decision: Accept