Title: Product Distribution Learning with Imperfect Advice
Venue: NeurIPS 2025
Accepted: yes
Keywords: distribution learning

statistics

algorithms

sample complexity
Abstract: Given i.i.d.~samples from an unknown distribution $P$, the goal of distribution learning is to recover the parameters of a distribution that is close to $P$. When $P$ belongs to the class of product distributions on the Boolean hypercube $\{0,1\}^d$, it is known that $\Omega(d/\epsilon^2)$ samples are necessary to learn $P$ within total variation (TV) distance $\epsilon$. We revisit this problem when the learner is also given as advice the parameters of a product distribution $Q$. We show that there is an efficient algorithm to learn $P$ within TV distance $\epsilon$ that has sample complexity $\tilde{O}(d^{1-\eta}/\epsilon^2)$, if $\|\mathbf{p} - \mathbf{q}\|_1<\epsilon d^{0.5 - \Omega(\eta)}$. Here, $\mathbf{p}$ and $\mathbf{q}$ are the mean vectors of $P$ and $Q$ respectively, and no bound on $\|\mathbf{p} - \mathbf{q}\|_1$ is known to the algorithm a priori.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper falls in the line of research of algorithms with predictions where algorithms are equipped with advice or predictions. Given recent developments in machine learning it is often too pessimistic to run a worst-case algorithm and not leverage predictable structures of the input. The line of research seeks to bridge algorithms and ML by obtaining provably improved algorithms with accurate predictions while also maintaining worst case guarantees.\n\nThe paper studies the problem of learning a product distribution $\\bf{p}$ (parametrized by Bernoulli parameters $(p_1,\\dots,p_d)$) over the hypercube $\\{0,1\\}^d$. It is well-known that $\\Theta(d/\\varepsilon^2)$ is necessary and sufficient in the classic setting of the problem where we want to return a distribution of total variation distance at most $\\varepsilon$ to $\\textbf{p}$. The paper shows that if each $p_i\\in [\\tau,1-\\tau]$ (they call this assumption \\emph{balancedness}), say for some $\\tau=\\Omega(1)$, if we are given access to an \\emph{advice} distribution $\\bf{q}$ such that $\\|\\textbf{p}-\\textbf{q}\\|_1$ is small, one can reduce the sample complexity. Specifically, one can obtain sublinear sample complexity $\\tilde O (d^{1-\\eta}/\\varepsilon^2)$ as long as $\\|\\textbf{p}-\\textbf{q}\\|_1\\leq \\varepsilon d^{0.5-\\Omega(\\eta)}$. The paper additionally shows lower bounds demonstrating that without the balancedness assumption, even with very accurate advice, say $\\|\\textbf{p}-\\textbf{q}\\|_1=O(1)$, we must use $\\Omega(d/\\varepsilon)$ samples. Additionally, they show that when $\\|\\textbf{p}-\\textbf{q}\\|_1\\geq \\varepsilon\\sqrt{d}$, we again require $\\Omega(\\sqrt{d})$ samples.\n\nI found the problem well motivated and the techniques quite interesting. The main challenge is to estimate $\\|\\textbf{p}-\\textbf{q}\\|_1$ using a small number of samples from $\\textbf{p}$. If this quantity is large, say $\\geq \\lambda$, the algorithm just resorts to its classic counterpart which returns the empirical mean of a bunch of samples. Otherwise, they can return the solution to a constrained (by $\\|\\hat \\textbf{p}-\\textbf{q}\\|\\leq \\lambda$) least squares problem. With this constraint, the sample complexity is correspondingly smaller.\n\nSTRENGTHS AND WEAKNESSES:\n(1) The paper is well and clearly written and I had little difficulty following the proofs.\n(2) Distribution learning is an important problem and I found the  model well-motivated and interesting.\n(3) The algorithmic/theoretical content of the paper is quite nice. It seems to me that the main algorithmic contribution is the algorithm \\texttt{ApproxL1} which fails if $\\|\\textbf{p}-\\textbf{q}\\|_2$ is large (which implies that $\\|\\textbf{p}-\\textbf{q}\\|_1$ is large) and otherwise outputs an estimate $\\lambda$ of $\\|\\textbf{p}-\\textbf{q}\\|_1$.\n\n\\textbf{Weaknesses} (1) The author's state that their paper and analysis parallels that of [BCJG24] which address the problem of learning Gaussian distributions with imperfect advice. They state that one of their different contributions is their new $\\ell_2$-approximation algorithm. If by this, they refer to the one of Lemma 3.2, this is maybe a little disappointing as the ideas are very similar to [DK16]. I am thus curious whether the ideas behind Lemma 3.3 are new, or if similar ideas appeared in [BCJG24].\n\nQUESTIONS:\nl89: You say that we can simply return $\\textbf{q}$ but this sound different from what your algorithm does, namely either running the least squares algorithm or the classic algorithm. Can you clarify?\n\nl155: $\\|\\textbf{p}\\|\\to \\|\\textbf{p}\\|_1$\n\nl159: $m(k,\\alpha, \\delta')$ hasn't been defined.\n\nAlgorithm 1, l6: Do  you want to define $\\mathcal{S}_j$ as a multiset?\n\nAlgorithm 1, l11: It's not really clear what it means to run the algorithms with these parameters. For example, you should mention that $l_i$ is the accuracy paramter.\n\nl174: $n_{k,\\varepsilon}$ and $r_\\delta$ haven't been defined.\n\nl181: I think you want to claim that $\\|\\textbf{p}\\|\\to \\|\\textbf{p}\\|_2$ is large\n\nl187: What does it mean to perform a constrained LASSO?\n\nAnd finally: The dependence on $\\varepsilon$ seems to be $1/\\varepsilon$ in your lower bound in Lemma 4.2, but the upper bound is $O(d/\\varepsilon^2)$. Do you have any idea what the right dependence is?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors considered the problem setting of learning product distribution under imperfect advice, \nWith some assumption on the mean for each coordinate that is balanced (not too close to 0 or 1).\nThe problem setting is as follows.\nGiven an unknown product distribution of {0,1}^d (with p be the mean vector of it), \nand advice vector q.\nThe algorithm needs to learn a $\\hat{p}$ as an estimation of p, such that the product distribution specified by $\\hat{p}$\nIt is close to the true input product distribution in total variance distance. \n(With the balanced assumption here, this is equivalent to small l_2 norm between $\\hat{p}$ and $p$.)\nIf the given advice is good ($\\|q-p\\|_1$ is small), the algorithm needs to use fewer samples than the trivial $d/\\epsilon^2$ sample complexity.\nAnd if the given advice is bad, the algorithm needs to perform as well as the trivial $d/\\epsilon^2$ sample complexity. \nThis need to be done without the previous knowledge of $\\|q-p\\|_1$.\n\nFor their result, the authors give a polynomial time algorithm with $d^{1-\\eta}/\\epsilon^2$ sample complexity algorithm when $\\|p-q\\|_1<\\epsilon d^{0.5-\\Omega(\\eta)}$.\nThe high-level idea they used is the following:\nThe first step is to get an estimation of $\\|q-p\\|_1$.\nDirectly approximating it would result in linear sample complexity and defeat the purpose.\nSo the author partitions it into d/k blocks, each with k coordinates.\nThen, estimate the L2 norm difference on each block and add them up.\nThis gives an estimation of the l1 norm by a multiplicative error factor of at most $\\sqrt{k}$.\nAfter that, the problem can then be reformulated as the convex optimization below line 84.\n\nI feel like the result is far from optimal, and a lot more should be done in this setting. \nIn particular, there isn't much justification from the lower bound side, or some argument such as\nthis result is the best achievable under some particular type of methods. \nFurthermore, it seems both the assumption and the guarantee are very artificial. \n(mostly to incorporate the methodology using L2 norm from my understanding)\nThe author didn't explain about this L2 norm approach, like whether they are typical in this setting, or if this is the best we can do.\nThis is based on my limited knowledge of this topic, and I would encourage the authors to point out any mistakes I may have made.\n\nSTRENGTHS AND WEAKNESSES:\nOne weakness I can think of is that the balanced assumption isn't very well justified.\nWhile the author did mention that there no sublinear sample complexity algorithm even when advice is O(1) close \nto the truth in l1 distance, it is unclear how sample complexity changes with o(1) l1 distance.\nAnd it is plausible that less samples are needed given the previous $\\sqrt{n}/\\epsilon^2$ indentity testing result.\n\nQUESTIONS:\nCan the authors explain more about why the assumptions are need here?\nEspecially if there is anything known about the sample complexity for unbalanced product distribution\nwhen the advice has o(1) l1 error.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies distribution learning with advice, within the line of research on learning-augmented algorithms or algorithms with predictions. In the standard version of the problem, the algorithm is given samples from a product distribution $P$ over $\\{0,1\\}^d$ and the goal is to output an estimate of $P$ which is $\\epsilon$ close in TV distance. In this work, the algorithm is also given access to the parameters of another distribution $Q$, also a product distribution over $\\{0,1\\}^d$. The goal is to design an algorithm which is more sample efficient if $Q$ is close to $P$ and which is not much more sample inefficient ($O(d/\\epsilon^2)$ samples is the benchmark in the standard setting) for worst-case $Q$.\n\nThe main result of this paper is an algortihm which improves over $O(d/\\epsilon^2)$ samples as long as $\\|Q-P\\|_1 \\ll \\epsilon \\sqrt{d}$. This result is restricted to the case where every coordinate of $P$ has probability bounded away from $0$ and $1$. The authors show that both this condition and the dependence on the $\\ell_1$ distance between $P$ and $Q$ are required: there exist distributions which do not satisfy these conditions for which sublinear in $d$ samples are insufficient for learning.\n\nIf the $\\ell_1$ distance between $P$ and $Q$ was known, then the authors show that a reduction to hypothesis selection by covering the $\\ell_1$ ball around $Q$ with $\\ell_2$ balls of radius $\\epsilon$. To estimate the distance, the authors develop a tolerant tester for product distributions based on tolerant testers for discrete distributions over $[d]$.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper addresses the natural and fundamental problem of learning product distributions over the Boolean hypercube. The advice in the model is natural and the main result shows a nice tradeoff between the accuracy of the prediction and sample complexity of the algorithm.\n\nThe dependence on the $\\ell_1$ norm and the restriction to balanced distributions are justified by hard examples. Despite this, the balanced assumption in particular seems somewhat strong. It would be very interesting if some relaxations of this assumption could be made which avoid the hard example and still allow for sublinear learning when the prediction is good.\n\nThe paper is purely theoretically, and I would be very interested to see (proof-of-concept) experiments which could perhaps investigate whether $\\ell_1$ distance between $p$ and $q$ is strongly correlated empirically with the accuracy of perhaps a simplified version of your tester.\n\n## Minor Comments\n\nLemma 4.2 is missing a period at the end of the lemma.\n\nQUESTIONS:\nIs there hope to improve the quantitative dependence on the $\\ell_1$ distance?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nI really like the model of \"learning with imperfect advice\", which I think is both conceptually interesting and practically well motivated.  In my own words, and at a high level,  this model is the following: A learner interacts with some untrusted third party which gives them \"advice\". If the advice is \"good\" then the query complexity of the learner should improve over the baseline learning algorithm. However, the learner should never require more queries than the baseline algorithm, even if the advice is \"bad\" -- i.e. the learning algorithm should be robust against \"bad\" advice. The learner has no promise on whether the advice is good or bad.\nGiven this setup, there is a clear high-level algorithm for this problem, which is roughly the following:\nTest if the advice is good or bad.\nIf the advice is good, accept the advice - you're done.\nIf the advice was bad, revert to learning, possibly using extra information gained during testing.\nFrom the above its clear that this model mixes learning and testing in an interesting way,  and that learning with sublinear query complexity should be possible when the advice is good and a sublinear test for this is possible. As before, I think this really is an interesting model, and that there are many many natural and interesting open questions.\nWith this in mind, the submitted manuscript studies the problem of learning product distributions in this model, where the advice is a full description of a candidate product distribution. This is a natural analogue of previous work in this area on learning multivariate Gaussians with imperfect advice (BCGJ24), and indeed the high-level structure of the algorithm given in the manuscript is analogous to the one given in BCGJ24. However, to instantiate this algorithm the authors of the submitted manuscript need to develop a novel tolerant mean tester and l1 approximation algorithm. Interestingly, it also turns out that unlike the multivariate gaussian case, one provably needs a \"balancedness\" assumption in this setting (even though this assumption is not necessary for identity testing of product distributions!). With this in mind, I think the manuscript has the following strengths and weaknesses:\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nI think that the model considered is very interesting and well motivated, and the paper provides some foundational results in this model, which has only very recently been proposed and studied for the first time. I think its great to draw more attention to this model!\nThe paper exposes an interesting and unexpected  discrepancy between testing product distributions and learning product distributions with imperfect advice.\n\nWeaknesses:\nIn many ways (both upper and lower bounds) the submitted manuscript parallels the earlier work of BCGJ24 on learning multivariate gaussians in this model. While the authors do require some novel elements -- namely a new tolerant mean tester and l1 approximation algorithm -- the high level ideas mostly already present in earlier work. To be fair though, because there is an \"obvious\" high-level algorithm, most work in this area will probably share the same key ideas and methods.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}