Title: Distribution Learning Meets Graph Structure Sampling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Distribution learning

Bayesian networks

Chordal graphs

Online learning
Abstract: This work establishes a novel link between the problem of PAC-learning high-dimensional graphical models and the task of (efficient) counting and sampling of graph structures, using an online learning framework. The problem of efficiently counting and sampling graphical structures, such as spanning trees and acyclic orientations, has been a vibrant area of research in algorithms. We show that this rich algorithmic foundation can be leveraged to develop new algorithms for learning high-dimensional graphical models.

We present the first efficient algorithm for (both realizable and agnostic) learning of Bayes nets with a chordal skeleton. In particular, we present an algorithm that, given integers $k,d > 0$, error parameter $\varepsilon > 0$, an undirected chordal graph $G$ on $n$ vertices, and sample access to a distribution $P^\ast$ on $[k]^n$; (1) returns a Bayes net $\widehat{P}$ with skeleton $G$ and indegree $d$, whose KL-divergence from $P^\ast$ is at most $\varepsilon$ more than the optimal KL-divergence between $P^\ast$ and any Bayes net with skeleton $G$ and indegree $d$, (2) uses $\widetilde{O}(n^3k^{d+1}/\varepsilon^2)$ samples from $P^\ast$ and runs in time $\mathrm{poly}(n,k,\varepsilon^{-1})$ for constant $d$. Prior results in this spirit were for only for trees ($d=1$, tree skeleton) via Chow-Liu, and in the realizable setting for polytrees (arbitrary $d$ but tree skeleton). Thus, our result significantly extends the state-of-the-art in learning Bayes net distributions. We also establish new results for learning tree and polytree distributions.


Peer reviews:
comments: SUMMARY:
This paper uses ideas from online learning to make progress on PAC learning of Bayes networks.
They present an efficient (agnostic) PAC learning algorithm when the Bayes networks has a known chordal skeleton and improve in several ways in the case where the graph is a known tree which was studied in recent work [11].

STRENGTHS AND WEAKNESSES:
The idea of treating different orientations as different experts is definitely attractive. 
Similarly improving the efficiency by using conditional independence and estimate conditional probabilities from samples is very natural. 

Of course restricting to a known graphical structure where only the orientation is unknown is theoretically interesting but maybe hard to motivate from applications?

The ideas of the proofs are a bit too high level and I am not sure how confident I am in their correctness.

QUESTIONS:
While the high level picture is clear - I am not quite sure I see the argument but this may be due to space constraints. 
In particular it would be good to at least hint at the body of the paper as to: 
1. Where do you use the fact that the graph is chordal? 
2. It is not clear how local estimates are sufficient to compute the probability of each particular orientation.
3. Moreover, given that you consider potentially unbounded degree graphs, it is not clear how much data do you need in order to estimate the local weights especially as these need to be very accurate to apply randomized weighted majority.

LIMITATIONS:
The main limitation is the sparsity of the arguments provided. There is just not enough to convince that it works out.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the PAC-learning of Bayesian network distributions by reducing the problem to online expert learning over all DAGs with a given chordal skeleton. The authors propose treating each possible DAG as an expert, aggregating them via efficient combinatorial counting of acyclic orientations. They prove that for a chordal skeleton with max indegree d, their algorithm learns a distribution within ε in KL-divergence using $$Õ(n³k^{d+1}/ε^²)$$ samples, extending classic Chow–Liu tree and polytree results.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**  \n- **Novel theoretical bridge:** Clever reduction of Bayes net learning to online no-regret expert aggregation using combinatorial graph algorithms.  \n- **Generalization:** First efficient agnostic PAC learner for chordal-skeleton Bayes nets, subsuming tree and polytree cases.  \n- **Rigorous bounds:** Provable sample complexity and runtime guarantees grounded in well-established online learning theory.  \n\n**Weaknesses**  \n- **Known skeleton assumption:** Sidesteps the hardest part—structure discovery—raising questions about practical usability when the skeleton is unknown or estimated.  \n- **No experiments:** Lacks any empirical validation or simulations to gauge performance, leaving constants and scalability unclear.  \n- **Misleading framing:** Title mentions “latent confounder” and causality, but the work focuses purely on distribution learning under a known graph, which may confuse readers.\n\nQUESTIONS:\n1. **Skeleton acquisition:** How would one obtain a correct chordal skeleton in practice? Would skeleton estimation errors undermine the guarantees?  \n2. **Unknown structure:** Can the online framework be extended to learn the skeleton itself (beyond the tree case), or is that inherently intractable?  \n3. **Causal interpretation:** Please clarify the role of “latent confounders” in this setting—does your method handle hidden-variable models, or is it purely observational?  \n4. **Empirical feasibility:** Have you attempted small-scale simulations (e.g., n≈20) to evaluate sample complexity and runtime in practice?\n\nLIMITATIONS:\nThe authors do not experimentally assess limitations; a discussion of skeleton errors, runtime constants, and potential failure modes would be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel distribution learning algorithm for graphical models. The proposed method is the first to enable PAC learning of bounded indegree chordal-structured distributions using only the skeleton of the graphical model as prior information. It can also be applied to learning polytree-structured and tree-structured distributions, and is shown to outperform existing methods theoretically. The key idea behind the proposed approach is to leverage algorithms from the online learning literature, such as the Exponential Weighted Average algorithm and the Randomized Weighted Majority algorithm. By exploiting structural properties of graphical models, the method enables efficient sampling of experts within the algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nAs I am not an expert in the learning theory of graphical models, I may not be able to accurately assess the validity or correctness of the authors’ claims. I have not wholly followed all the proofs provided in the appendix.\n\n# Strengths\n\n- The proposed method is innovative and elegant. It applies techniques from online learning to estimating distributions over graphical problems and suggests promising directions for future research.\n- The method is extensively analyzed from a theoretical perspective, and improvements over existing approaches are clearly demonstrated.\n\n# Weaknesses\n\n- Certain assumptions limit the applicability of the proposed method, such as the requirement that knowledge of the graphical model's skeleton and the distribution's chordal structure. \n- The paper has a complex structure, making it difficult to follow. Understanding the overall framework requires reading the appendix in detail. While key components necessary for understanding the method are deferred to the appendix, some arguably less critical content (e.g., the paragraph \"Why KL divergence?\") remains in the main text. The organization between the main body and the appendix warrants reconsideration.\n\nQUESTIONS:\n- While the proposed method relies on several assumptions—such as the distribution being a bounded indegree chordal-structured distribution and the skeleton being known—can these limitations be addressed or relaxed through extensions of the proposed framework?\n- In the context of practical applications, to what extent do the assumptions made by the proposed method (e.g., known skeleton, chordal structure) pose a barrier? Are they relatively benign, or could they significantly hinder real-world deployment?\n\nLIMITATIONS:\nThe proposed method makes various assumptions about the shape and input of graphical models, limiting the problems to which it can be applied.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives computationally efficient on-line learning algorithms for certain families of Bayesian Networks, models with a known skeleton with a chordal graph of bounded in-degree, and trees. A computationally efficient algorithm for trees was known (indeed, the algorithm itself, by Chow and Liu, is very old) but the proposed method obtains optimal sample complexity -- the dependence on the size of the range of the random variables is improved to quadratic, rather than cubic. The method yields guarantees for agnostic improper learning of such structures. (A further extension to learning Bayes Nets with a moralization with a O(1)-size vertex cover, using another prior work for sampling, is described in an appendix.)\n\nThese results are obtained by leveraging the following simple observation: the multiplicative updates used in existing on-line learning methods can be distributed across the factors of the probabilistic model. This is essentially equivalent to running a separate learner for the conditional probability tables for each of the factors; this corresponds to a mixture of models that can be sampled by independently sampling a table for each factor.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper has two main strengths. The first is that the underlying idea is very clean and well-suited to these kinds of probabilistic models. The second is that it indeed obtains quantitatively improved guarantees for specific problems, like learning tree-structured distributions.\n\nThe main weaknesses are first, that the approach relies on running the learner on a discretization of the conditional probability tables. Although the conditional probability tables themselves are inherently exponential in the number of assignments to the parents, and such a dependence is to be expected, the requirement to maintain parameters for the individual table values seems somewhat impractical. Second, the chordal graph learner relies on possession of a known skeleton for the DAG, and this approach doesn't address the structure learning problem that poses the more fundamental barrier to the use of probabilistic graphical models in practice. It is a theoretical improvement -- I am not aware of a method that is guaranteed to learn such structures -- but it is not clear if it makes a difference in practice.\n\nThe \"novel link\" between learning graphical models and sampling structures, described in the abstract, seems like an over-sell. The work uses sampling methods for improper learning of the models, but this is (just) an application, not the discovery of a deeper connection between the problems. (For example, the paper does not seem to establish a connection in the other direction.)\n\nQUESTIONS:\nDo you have a deeper link between counting/sampling of graph structures and learning graphical models? (Besides the application here, and the casting of the computation of partition functions as a counting problem; I mean, is there a tight link between these problems, as the abstract and introduction seems to suggest? Where is it spelled out in the paper?)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}