Title: Learning Juntas under Markov Random Fields
Venue: NeurIPS 2025
Accepted: yes
Keywords: pac learning

learning juntas

smoothed analysis

ising models

markov random fields
Abstract: We give an algorithm for learning $O(\log n)$ juntas in polynomial-time with respect to Markov Random Fields (MRFs) in a smoothed analysis framework, where only the external field has been randomly perturbed.  This is a broad generalization of the work of Kalai and Teng, who gave an algorithm that succeeded with respect to smoothed *product* distributions (i.e., MRFs whose dependency graph has no edges). Our algorithm has two phases: (1) an unsupervised structure learning phase and (2) a greedy supervised learning algorithm. This is the first example where algorithms for learning the structure of undirected graphical models have downstream applications to supervised learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives an efficient PAC learner for learning $O(\\log n)$-sized juntas when the examples are drawn from a Markov Random Fields. This result extends the setting of learning Boolean concept classes from smoothed product distributions (Kalai and Teng'08).\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\nThe novelty of this paper lies in connecting two important subfields of learning - learning graphical models and learning Boolean concept classes, and providing tools for analysis in this setting.\n\n# Weaknesses\n\nIn light of existing results in the literature, the paper does a poor job of motivating why the $(O(1), O(1))$-smoothed MRF setting is important or why Theorem 1.3 is interesting since we are constrained to learning only $O(\\log n)$-sized juntas over slightly perturbed MRFs. The paper is also quite poorly presented and there is an extreme lack of clarity in the non-technical sections. Please the questions section for more details.\n\nQUESTIONS:\n1. Line 3, 44: The usage of the term _broad generalization_ in the context of Kalai and Teng (2008) may be misleading. $O(\\log n)$-sized juntas are a subclass of size-$n$ decision trees, and the results in this paper do not seem to straightforwardly extend to learning of decision trees. I would invite the authors to at least add a footnote clarifying this issue.\n\n2. Line 20: While learning under different marginal distributions is an important question in its own right - and merits standalone discussion, it is important to note that most prior work focused on learning juntas actually consider the setting of learning under label noise. Curiously, any reference to this entire line of learning and testing juntas is missing from the paper. This should be explicitly addressed in the work (at least with a few notable references), especially considering that the paper is considering an extremely classical problem that has been around in the literature for over three decades.\n\n3. Section 1.1, Lines 30-33: How does $c$ relate to $\\Delta$ or $\\sigma$? What is $n$ in this context?\n\n4. Line 38: If I were to understand the earlier setup, the joint distribution over the instances is a product distribution. Since we are in the noiseless setting (assuming no label noise), what exactly are we referring to as the marginal distribution in this case?\n\n5. Definition 1.1, Line 51 onwards: The definition needs to be heavily rewritten simply based on the fact that this is the _only_ definition (formal or informal) for undirected graphical models, and the venue of choice is NeurIPS, which is a general purpose machine learning conference, unlike specialized theoretical venues like COLT.\n\n- How do we define a graph $G$ in this context? Are we still considering finite simple graphs? What is the degree $d$ of $G$?\n- In the context of this paper, can we consider $G$ to simply be the Boolean hypercube?\n- In equation $1$, how exactly is $\\psi_S$ defined? In Line 61, the authors note that the linear part of the polynomial $\\psi$ is called the external field. What is the _linear part_ of $\\psi=\\sum_S \\psi_S$ in this context? What happens if $\\psi_S$ is always non-linear - does that indicate an absence of an external field?\n\n6. Definition 1.2: I believe it needs to be explicitly stated that $D$ is an undirected graphical model.\n\n7. Line 73-74, 79-80: Sources should be cited, or an argument needs to be provided. How is this a generalization of p-biasedness?\n\n8. Line 75, 76: In the absence of normalization constants everywhere, there should be short derivation of the expression for the multiplicative perturbation of the density function. Once again it is ambiguous what perturbing the external field exactly means here.\n\nLIMITATIONS:\nIt is unclear if the results or techniques in this paper imply anything non-trivial about learning more structured concept classes such as Decision Trees. Can the authors provide experimental evaluations indicating the difficulty of learning juntas in this setting compared to the i.i.d. setting and the smoothed product setting?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a polynomial-time algorithm to learn a \"junta\" from data generated by an MRF. In this framework, juntas should be understood as classifiers that implement the irrelevance of certain features. The work represents an extension of a previous work roughly corresponding to fully factorised distributions (= MRF with disconnected nodes). This is a technical paper with detailed proofs in the main body, but it completely lacks empirical validation.\n\nSTRENGTHS AND WEAKNESSES:\n- The extension from the fully factorised case to that of an MRF, under some reasonable assumptions on the resulting joint distribution, is a clear advance in the SOTA.\nYet, the paper seems to focus on the application of the result to machine learning, but there are almost no efforts to highlight the real potential of the result in ML. In this perspective, the lack of any experiment is an evident weakness,  which is, in my view, sufficient to justify a negative recommendation and a low significance (for this particular conference).\nIndeed, I could imagine an ML application involving a classification dataset, with the junta playing the role of a feature selection mechanism paired with a classifier. Yet, I doubt whether this nice approach would lead to competitive results: we know from that literature that approaches based on logical formulae are typically sub-optimal in modern ML.\n\nQUESTIONS:\nAs mentioned, my main issue with the paper is the lack of a clear connection with machine learning and the (consequent?) absence of any experimental analysis. My main question for the authors is whether they considered this link straightforward and decided to keep it implicit, or whether they believe that the result in the paper has an interest by itself from a learning theory perspective, with no need to establish a connection with machine learning.\n\nI would also like to know whether the choice of ending the paper with the statement of the theorem (and its proofs), and without conclusions, reflects the idea that the technical results do not require any additional commentary.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper investigates the following problem: Given random samples i.i.d. drawn from a Markov Random Field (with a known dependency graph of degree $O(\log n)$), under a smoothed analysis framework, and labeled by an unknown k-junta, is there an efficient algorithm to recover the underlying k-junta? This work represents a significant generalization of prior research, which was largely confined to product distributions. A key contribution is the introduction of a conditional correlation statistic, $I(i,\rho)$, which effectively leverages the property of MRFs to distinguish relevant features from irrelevant ones. Building on this statistic, the authors develop an algorithm with both runtime and sample complexity bounded by $n^{o(k)}$.

STRENGTHS AND WEAKNESSES:
Strengths:
A nice and natural generalization of product distribution in terms of MRF beyond worst-case analysis

Weaknesses:
As the authors themselves acknowledge, a notable limitation of the proposed method is its reliance on the assumption that the MRF’s dependency graph is known. In the absence of this knowledge, one would need to resort to structure learning techniques for bounded-degree MRFs, incurring substantial additional computational and sample complexity.

QUESTIONS:
The method is designed specifically for learning k-juntas, which is a relatively restricted model. It would be helpful if the authors could clarify what makes it particularly challenging to recover more complex concept classes, such as k-depth decision trees or k-DNFs under MRF distributions.

In line 162, the lowercase variables  s and t appear to be undefined in the surrounding context, this seems to be a typographical error.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-02

comments: SUMMARY:
This paper studies a theoretically deep question: the $k$-junta problem where marginal distributions are perturbed instead of uniform over the hypercube; in particular, the problem in this paper is over Markov random fields (MRF). The main result yields a successful two-phased algorithm (with high probability) with sample size $\Omega(\text{poly}(\log n, \exp(\lambda(d+k)), 2^{d+k}, \sigma^{-k}, 1/\delta))$. Their result is a natural generalization of earlier works for smoothed product distributions.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The generalization is natural and significant: Extending the problem scale to a broader class of distributions is a long standing open problem, and the authors gave an elegant solution and sound analysis.
2. The techniques illustrated in Section 3 is theoretically rich and novel. It is refreshing by blending unsupervised and supervised methods. The algorithm, despite requiring a possibly high sample size (see Weakness), will very likely stimulate further research on this area and further optimization.

Weakness:
My major concern is the size of $k$; it is only logarithmic and therefore may restrict the paper's scope. I understand that this constraint is necessary to bound sample complexity to polynomial, but, to the best of my knowledge, problems in similar themes ($k$-parity, $k$-junta, etc) usually assume $k$ to be a constant fraction of $n$.

QUESTIONS:
1. Please see Weakness
2. Does the learning of $k$-junta problem have any relevance with the task of learning $k$-parity, or do results for one problem inspire the other? If the answer is yes, then discussions for $k$-parity learning may be helpful.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03