A Black-Box Debiasing Framework for Conditional Sampling
Authors: Han Cui
Jingbo Liu
Venue: NeurIPS 2025 poster ()
Keywords: Conditional sampling
Bayesian Inference
Posterior Approximation
Bias Correction
Primary area: probabilistic_methods

Abstract: Conditional sampling is a fundamental task in Bayesian statistics and generative modeling. Consider the problem of sampling from the posterior distribution $P\_{X|Y=y^\*}$ for some observation $y^\*$, where the likelihood $P\_{Y|X}$ is known, and we are given $n$ i.i.d. samples $D=\\{X\_i\\}\_{i=1}^n$ drawn from an unknown prior distribution $\pi\_X$. Suppose that $f(\hat{\pi}\_{X^n})$ is the distribution of a posterior sample generated by an algorithm (e.g. a conditional generative model or the Bayes rule) when $\hat{\pi}\_{X^n}$ is the empirical distribution of the training data. Although averaging over the randomness of the training data $D$, we have $\mathbb{E}\_D\left(\hat{\pi}\_{X^n}\right)= \pi\_X$, we do not have $\mathbb{E}\_D\left\\{f(\hat{\pi}\_{X^n})\right\\}= f(\pi\_X)$ due to the nonlinearity of $f$, leading to a bias. In this paper we propose a black-box debiasing scheme that improves the accuracy of such a naive plug-in approach. For any integer $k$ and under boundedness of the likelihood and smoothness of $f$, we generate samples $\hat{X}^{(1)},\dots,\hat{X}^{(k)}$ and weights $w\_1,\dots,w\_k$ such that $\sum_{i=1}^kw_iP\_{\hat{X}^{(i)}}$ is a $k$-th order approximation of $f(\pi\_X)$, where the generation process treats $f$ as a black-box. Our generation process achieves higher accuracy when averaged over the randomness of the training data, without degrading the variance, which can be interpreted as improving memorization without compromising generalization in generative models.

A Black-Box Debiasing Framework for Conditional Sampling
Han Cui
Jingbo Liu
University of Illinois at Chhama-Champaign
University of Illinois at Urbana- trtana-Champaign Champaign, IL jingbol@illinois.. edu
Champaign, IL hancui58illime www.::illliiiii...mmmm edu
Abstract
Conditional sampling is a fundamental task in Bayesian statistics and genera- tive modeling. Consider the problem sampling from the posterior distribution Pxjy=v' for some observation y". where the likelihood Py is known, and we are given n i..d. samples D = {X+). drawn from an unknown prior distribution x. Suppose that (++x*) the distribution of posterior sample generated by an algorithm (e.g. conditional generative model or the Bayes rule) when x is the empirical distribution of the training data. Although averaging over the randomeess of the training data D, we have ED (fx.) wx, we do not have ED (f(xx)) x = (xxx due to the nonlinearity of leading to a bias. this paper we propose black-box debiasing scheme that improves the accuracy of such a naive plug-in approach. For any integer and under boundedness of the likelihood and smoothness of f, we generate samples { . (A) and weights Wi...., W2 such that Σ Px is a k-th order approximation of f(Tx), where the generation process treats higher accuracy when averaged over the randomness of the training data, with- out degrading the variance, which can be interpreted as improving memorization without compromising generalization in generative models.
black-box. Our generation process achieves
Introduction
Conditional sampling is a major task Bayesian statistics and generative modeling. Given an observation y", the objective is to draw samples from the posterior distribution Px where the likelihood Pylx is known but the prior distribution я is unknown. Instead, we are provided with a dataset = {x,) consisting
i.i.d. samples drawn from xx-
The setting is common in a wide range of applications, including inpainting and image deblurring 5 (where X an image and Y|X a noisy linear transform), text-conditioned image generation (where X is an image and Y is natural language prompt), simulating biomedical structures with desired properties, and trajectory simulations for self-driving cars. Moreover, conditional sampling is equally vital in high-impact machine learning and Bayesian statistical methods, partic- ularfy under distribution shift, such as in transfer learning. For instance, conditional sampling has enabled diffusion models to generate trajectories under updated policies, achieving state-of-the-arn performance in offline reinforcement learning
7 7
X 26 Pseudo-labeling, a key technique for un- supervised pretraining [00 and transfer learning calibration [00] relies on generating conditional labels. Additionally, conditional diffusion models seamlessly integrate with likelihood-frre inference 6 18 27). Existing approaches often use generative models such as VAEs Diffusion models to generate samples by learning Px|y=y= implicitly from the data.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Our work focuses on approximating the true posterior Pxiy=y- using the observed samples D x" = (X1,..., .xx) and the new observation y", but without the knowledge of the prior. Denote by P - D the approximating distribution. We can distinguish two kinds of approximations: First, P = Px with high probability over D, which captures the generalization ability since the model must learn the distribution from the training samples. This criterion commonly adopted in estimation theory and has also been examined in the convergence analysis of generative models [16 28 26 22 Second, (
.D) R Px|Y=y' is a weaker condition since it only requires approximation when averaged over the randomness of the training data, but is still useful in some sampling and generative tasks, e.g. generating samples for bootstrapping Monte Carlo estimates of function expectations. The second condition captures the ability to memorize or imitate training sample distribution. It is interesting note that in the unconditional setting (i.e., without distribution shift), a permutation sampler can perfectly imitate the unknown training data distribution, even if 1, so the problem is trivial from the sample complexity perspective. However, in the conditional setting, is impossible to get such a perfect imitation with finite training data, as simple binary distribution example in Section 3.2 illustrates. It naturally leads the following question:
How fast can the posterior approximation converge to the true posterior as n + 00, and is there a sampling scheme achieving this convergence rate?
Contribution. We address the question above by proposing a novel debiasing framework for posterior approximation. Our main contributions can be summarized as follows:
Debiasing framework for posterior approximation. We introduce a novel debiasing framework for posterior approximation with an unknown prior. Our method leverages the known likelihood Pyjx and the observed data to construct an improved approximate posterior Px. (I yy) with provably reduced bias. In particular, let fffx"" represent the distribution of a posterior sample generated by an algorithm f when fx is the empirical distribution of the training data. Then for any integer k; assuming that the likelihood function
Py|x is bounded and f is sufficiently smooth, we generate samples X1), based on multiple resampled empirical distributions. These are then combined with designed (possibility negative) weights wi...., Wy to construct an approximate posterior:
A (k) from f
(-10)) = Σ we Px
which a k-th order approximation of (TT), treating the generation process I as black- box. Our generation process achieves higher accuracy when averaged over the randomness of the training data, but not conditionally on the training data, which highlights the trade-off between memorization and generalization in generative models. Specifically, we do not assume any parametric form for the prior and our method can achieve a bias rate of O(n -k) for any prescribed integer i and variance rate of O(n ))
Theoretical bias and variance guarantees. We establish theoretical guarantees on both bias and variance for the Bayes-optimal sampler under continuous prior setting and for broad class of samplers f with a continuous 2k-th derivative, as specified in Assumption 2 under the discrete prior setting. The proposed debiasing framework can also be applied in a black-box manner (see Remark 2 for the intuition), making it applicable to a broad class of of art conditional samplers, such as diffusion models and conditional VAE. Based on this perspective, we treat the generative model f as black box that can output posterior samples given resampled empirical distributions. Applying f multiple recursive resampled versions of the training data and combining the outputs with polynomial weights, we obtain bias-corrected approximation of the posterior. The procedure described in Algorithm|] I
Our approach is also related to importance sampling. Since the true posterior Pxly is intractable to compute, we can use expectations under the debiased posterior Px- (xly") ) to approximate the expectations under the true posterior P For test function h, we estimate Exxy--,- {h(X)) by
N
Px- (X,11") h m(X,)
Ep Ex (M(X)) RJ
(1)
N
q( ((,l))))
2

---

Algorithm Posterior Approximation via Debiasing Framework
Input: Observation y², likelihood Pylx. data X = (Xi,.... Xa), number of steps k, black-box conditional sampler f (i.e., a map from a prior distribution to a posterior distribution)
Output: X(), = =l,..., k such that Σjn (-1) P Pxo+t is high-order approximation of
the posterior Px|Y==*
1: Initialize (01) ;= #
2: for L 2 to k do
3: Generate ii.d. samples from f(f-1)
4: Let ((() be the empirical distribution of the sampled data
5: end for
6: for j 1 to & do
7: Generate samples X0) S(161)
8: end for
9: Return X(),j 1....,k
where X, ~ g(<ly")) for a chosen proposal distribution q. This resembles our method, in which approximate the true posterior by weighted combination Σ w; x And in ( the term (Xy'"") can be interpreted as a weight assigned to each sample, analogous to the
Px ///lx)//(XX, ly") weights w; in our framework. Therefore, expect that Algorithm can be broadly applied to Monte Carlo estimates of function expectations, similar to the standard importance sampling technique.
2 Related work
Jackknife Technique. Our work is related to the jackknife technique [17. a classical method for bias reduction in statistical estimation that linearly combines estimators computed on subsampled datasets. Specifically, the jackknife technique generates leave-one-out (or more generally, leave-s-out where 2 versions of an estimator, and then forms weighted combination to cancel lower-order bias terms. Recently, Nowozin [14] applied the jackknife to the importance-weighter autoencoder (IWAE) bound É.,, which estimates the marginal likelihood log π (I) using samples. While é. is proven to be an estimator with bias of order On 1). ) the jackknife correction produces a new estimator with reduced bias of order O(n Our paper introduces a debiaisng framework based on the similar idea that using a linear combination of multiple approximations to approximate the posterior.
Conditional Generative Models. Conditional generative models have become influential and have been extensively studied for their ability to generate samples from the conditional data distribution P(-ly) where y is the conditional information. This framework is widely applied in vision generation tasks such as text-to-image synthesis [13 24 where is an input text prompt, and image impainting 21 where y corresponds to the known part of an image. We expect that our proposed debiasing framework could work for broad class of conditional generative models to construct high order approximation of the posterior Pl-ly).
Memorization in Generative Models. The trade-off between memorization and generalization has been a focus of research in recent years. In problems where generating new structures or preserving privacy of training data is of high priority, generalization is preferred memorization. For example, a study by Carlini et al. demonstrates that diffusion models can unintentionally memorize specific images from their training data and reproduce them when generating new samples. To reduce the memorization of the training data, Somepalli et al. [19] applies randomization and augmentation techniques the training image captions. Additionally, Yoon al. (25) investigates the connection between generalization and memorization, proposing that these two aspects are mutually exclusive. Their experiments suggest that diffusion models are more likely generalize when they fail to memorize the training data. On the other hand, memorizing and imitating the training data may be intentionally exploited, if the goal is Monte Carlo sampling for evaluations of expected values, or if the task does not involve privacy issues, e.g. image inpainting and reconstruction. In these applications, the ability to imitate or memorize the empirical distribution of the training data becomes essential, especially when generalization is unattainable due to the insufficient data. Our work focuses

---

on memorization phase and shows that it is possible to construct posterior approximations with provably reduced bias by exploiting the empirical distribution.
Mixture-Based Approximation of Target Distributions. Sampling from a mixture of distributions a Px, 02,xx +... 0g Px, to approximate target distribution P is commonly used in Bayesian statistics, machine learning, and statistical physics, especially when individual samples or proposals are poor approximations, but their ensemble accurate. Traditional importance sampling methods often rely on positive weights, but recent work has expanded the landscape to include more flexible and powerful strategies, including the use of signed weights and gradient information. For example, Oates et al. [I5] uses importance sampling and control functional estimators to construct linear combination of estimators with weights aş to form
srriance-reduced estimator for an expectation under target distribution P Liu and Lee [12] select the weights dg by minimizing the empirical version of the kernelized Stein discrepancy (KSD), which often results in negative weights.
Problem setup and notation
Consider dataset {_ consisting of independent and identically distributed (i.i.d)) samples, where X X' drawn from an unknown prior distribution лх and the conditional distribution Pylx is assumed to be known. In the Bayesian framework, the posterior distribution of X given Y is given by
Px y (dx(y)y) = Prix(ul))Tx(dx)
1 (ylx(ule)Tx(de))
Given the observed data X (. .n)) and the new observation y", our goal is to approximate the true posterior Px]Y=
3.1
Naive plug-in approximation
natural approach is to replace the unknown prior xx with its empirical counterpart
+x- in the Bayes' rule to compute an approximate posterior which yields the plug-in posterior
dx.
Prix(V" (x))x=xxx=x)))))))))))))))) (dz)
(xix(dzz)y") (dzly")
(2)
J (x))tx) (dx)
Note that even though ED (fxx)) ) Rx, x the nonlinearity of Bayes' rule makes the resulting posterior (2) still biased, that is, ED P (xxir(-1')) * Pxiy-(ll""). the denominator in (2) were replaced with I (x)) (dx), then averaging the R.H.S. of 2 over the randomness in X would yield the true posterior Pxly (dxly")) = Py x (x)"x))=))))X x
Prix (v= (dr) exactly.
For typical choices of Prix which have nice conditional density (e.g., the additive Gaussian noise channel), J Pr (x) (dx) converges at the rate of n 1/2 11/2, by the central limit theorem. Con- sequently, ED P Px|Y=y.) converges to the true posterior at the rate ((n-1/2) in the oc-Renyi divergence metric regardless of the smoothness of πx. Under appropriate regularity conditions, can in fact show that E, (xxIX==)) converges at the rate of Ó(n which comes from the variance term in the Taylor expansion. Naturally, we come to an essential question: can we eliminate the bias entirely? That is, is it possible that Ed (- (xir(-l"")
3.2 Impossibility of exact unbiasedness
Exact unbiasedness is, general, unattainable. Consider the simple case where binary, that is. X Bern(g) for some unknown parameter q (0, 1). Define the likelihood ratio a(y")

---

Pyix(v"l))/Pr/xx (0). Then the true posterior is
aq
X = y" ~ Bern
aq +
On the other hand, if we approximate the posterior distribution as Pxir (1/y =y) = (per(p(k) upon seeing outcomes equal I, then
ED (xxIr Ep(k) p(k) (1) (1-
(3)
which is polynomial function of and hence cannot equal the rational function 0g/ (ag for all g. This implies that exact imitation, in the sense that Ep (P impossible. However, since a rational function can be approximated arbitrarily well by polynomials, this does not rule out the possibility that better sampler achieving convergence faster than, say, the O(n-1/2) rate of the naive plug-in method. Indeed, in this paper propose a black-box method that can achieve convergence rates as fast as On for any fixed k >> O.
(xyr(-ll"), VTT is
3.3 Objective and notation
Since the bias in the plug-in approximation arises from the nonlinearity of Bayes' rule, aim to investigate whether a faster convergence rate can be achieved. Our objective is to construct an approximation Px= (xryy v²) that improves the plug-in approximation by reducing the bias. Specifically, the debiased approximation satisfies the following condition:
Exn Px (zly= 5リ} )") - (x)r(<))")) <
Ex- {PxI((=))) (x)r(=l)"))
More generally, replace the Bayes rule by arbitrary map f from prior to posterior distribution (e.g. by generative model), and the goal is construct debiased map f such that
Ex= - -f(() | < <|Ex-=((xx>)) - ff(m)
Notation. Let 6, denote the Dirac measure, || |TTV denote the total variation norm. For any positive integer m, denote [m]] {I...., m) as the set of positive integers smaller than all equal to m. - Write b,, (a()) if bn, /a, is bounded by C(s)
Write 62 O(a,) ) if /a, aa bounded as n + oo for some constant C(s) that depends only on s. We use the notation a S to indicate that there exists constant C' > 0 such that S Cb. Similarly, Sx b means that there exists constant C(k) > that depends only on & such that a < C(k)b. Furthermore, for notational simplicity, we will use æ to denote the true prior x and á to denote the empirical prior fx in the rest of the paper.
4 Main result
4.1 Debiased posterior approximation under continuous prior
Let Ax denote the space of probability measures on X. Define the likelihood function {(x) == Pylx (xx())))) which represents the probability observing the data y given æ. Let f Ax + Ax be a map from the prior measure to the posterior measure, conditioned on the observed data y", Let B, be the operator such that for any function f Ax > Ax.
BrJ(p) ) = E (f()),
where ş denotes the empirical measure of i.i.d. samples from measure p.
We consider the case that f represents mapping corresponding the Bayes posterior distribution. Using Bayes' theorem, for any measure E Ax and any measurable set A c. X, the posterior measure f (TT) is expressed as
/ ((c)<((z))
[()) =
((x)m(d))
5

---

As discussed in Section 3 the equality = is not possible due to the nonlinearity of However, we can achieve substantial improvements over the plug-in method by using polynomial approximation techniques analogous to those from prior statistical work by Cai and Low 3 and Wu and Yang [3]. For 1, we define the operator Dr,k as a linear combination of the iterated operators B for O...., k
R-1
D
5-0
Assumption 1. The likelihood function È is bounded, i.e., there exists 0 < Li S L2 such that L1 S. ((xx) S L2.
The following theorem provides systematic method for constructing approximation of f(x)) with an approximation error of order On () for any desired integer k.
Theorem 1. Under Assumption
7 for any measurable set A C X and any k E N we have
Ex (Dn.ff()) - f(T)llTV = (n,
(5)
Varx" (D,,
O L2,
(6)
Remark 1. ( -I in (5) can be interpreted as weighted average of the distribution of some samples. Specifically, if we treat the coefficient (-1) as the weight Wy and (+t( as the distribution of some sample X6), then Dn,kI(#t= = -E Wxpx
Remark 2. Recall the binary case discussed in Section 3 illustrates that we cannol get an exact approximation for the true posterior. But from Bp. we demonstrate that even if 0 is impossible, it can be arbitrarity small. Although the the-
||Ex {Dn*f(ft)} f(x) oretical guarantees are derived for the Bayes-optimal sampler, (5) expected to hold for general sampler f such diffussion models. Here give the intuition for this conjecture. We view the operator (T)) = [+) as a good approximation of f(x). Lee, Bn x I, where I is the "small" operator. Under this heuristic, if (T) = O(n -1,, intuitively we have E f = O(n Using the bino- mial expansion of E = (1 -1-BB B,)* we have E* f(T) f(T) Σ,= - (-1)-
identity operator. This implies that the error operator E : I B,
f - E (- f(a)) -E the specific form of Dr,k-
This representation motivates
Remark 3. In general, the curse of dimensionality may arise and depends on the specific distri- bution of X and the likelihood function f. There no universal relationship between n and the dimension d. However, to build intuition, we give example that illustrates how n and d may relate. Suppose that Y = (Y(1).... Yd)) and X =
have i...d components, and
L & P(Y()) < L2 for i d. Then we have ((X) = P(YIX) E [T4. L Note that Lo in (5) can be bounded by LL// n for some constant C(k) depending only on To ensure that our debiasing method improves over the baseline method without debiasing
in the case of growing dimensions, it suffices to let n and d satisfy that (I.)/ (1/))4)))) 2.422k n & n- when
k which is equivalent kd & log(n).
Sketch proof for Theorem new operator
1 First let Sx. ((x))((xx and MA SA ((x))(dx) and introduce
Ca, = Σ 1=1
(0)--11 (-10-1B
then we have Dn, Cn,k- By the definition of B,, it suffices to show that
(n))((T((A) - (A) = In.
The first step is to express (T)) with the recursive resampled versions of the training data. Specifically, let (0)) n, (1) A and set (x ..... x( = (XI...,,,.). For j 1,...,k k.

---

we define #(5) the empirical measure of i.i.d. samples (x(-1)) ...., x)
drawn from the
measure # Additionally, let
Σ ((xx))
- m4 and 2 = n ) (xx3)
=
Then we have
(k) (-1) MA (1-1)
)
(n)
Σ ) (- (-1-- B = Σ
(7)
The second step is to rewrite 7 with Taylor expansion of - /(e))-1) + μ)) with respect to en e(i-1)
up order 2k 1. L1 ((x < L2 and Hoeffding's inequality implies that the expectation
of the residual term (e( (e((-11))a / / 1 for some & between en ()-1) + and is O La (n
Now we instead to show that
1 (-10-
11 (cl my = Olgdaadnn ") 'B
BET
J=1
since 7 equal to MA/P + 22h-11 (-1)" Bky L2,L2,
Define a new operator B : h + E(h(#)] for any : Ax + R and let h,(T) =
UA (d))) ((x))(r))) Then
BEr Σ (3) (-1) (1)
kk) ( (T). Σ
k
The last step prove
(1 - he (-) (TT) = ,T((T) + On (n (m=*). Finally ( follows
O Ln (n
(8)
since R is equivalent to Σ5= (4)
from the fact that (I- - B) hs (T) can be expressed finite sum of the terms which have the following form:
Qa,s,y ( ((())((()))
} II I (2)
where aa.s.vl S Cs(ss)n for some constnat (s) (see our Lemma
4.2 Debiased posterior approximation under discrete prior
In this section, we consider the case where X follows a discrete distribution. As mentioned in Remark 2 the result Theorem i is expected hold in broader class of samplers f under smoothness, extending beyond just the Bayes-optimal sampler f. The assumption of finite X' in this section allows us simplify some technical aspects in the proof.
Let the support of X be denoted as X' {u1, U2, Um }. Assume that | = m is finite, and X is distributed according value u; is given by 7 T( u;) = 4i for = 1, 2, m. Here, the probabilities are unknown and satisfy the usual constraints that y: 2 0 for all and E qi 1.
unknown prior distribution ा (2) such that the probability of X taking the
Let q (q1,.... 4m T represent the true prior probability vector associated with the probability distribution (z)). Let g be map from prior probability vector to a posterior probability vector. Then g(q) (g1 (q). ... Im (q)) is the probability vector associated with the posterior. Let T TT (Ti,..., Tm)) where T; = Ei= 1x,=uu for j 1,... m. In such setting, by the definition
of operator B,, we can rewrite the operator B, as
(x),(q) = E (g = Σ go
n2
q"
7

---

where Am f
Ej= V/ = n) and
n ()
n!
9' - qi Im
v1
Um 1
Additionally, let Am fqq E R o4 0, 9j
1} and let || 1-ll=*(smm. denote the CK (M(Am))
norm which is defined as || (A..) Σ salli Sk 1100 fl Dc for any f E (Am).
Assumption 2. |
is finite, and max Igsllere S for some constant G.
The following theorem provides systematic method for constructing an approximation of go (q) with an error of order O =) for any desired integer k.
qm)) .m be the true prior probability vector associated with a discrete probability distribution and T (Ti,,.., ,Tm) where T = Σi 1x,=u for j = 1, 1,,.. m. Under Assumption 2 the following holds for any E (1,-.. and any E N+:
Theorem 2. | m, let (q.....
Exp (DuN(g))(Tmm)) (q) = Ok,m Varx" (g.) = Ok.,,c(n-)))
Theorem 2
follows directly from the following lemma, which provides the key approximation result. Lemma 1. For any integers k, m E N+ and any function f E CAmm), we have
|| (f) fllo= = 11(B,, 1k/2/ (f) Sa.m IIfllc*(am) k/2
Note that Theorem holds for all mappings g that satisfy Assumption When g represents mapping corresponding to the Bayes posterior distribution, we know the exact form of g(g). Hence, we can explore sampling schemes for Bayes-optimal mapping g.
We claim that Bayes-optimal mapping g satisfies Assumption PyJx(v" (v"(u,) Using Bayes' theorem, the posterior probability of X us given y" is given by
2 In fact, let l, = ((us) =
6.90
Pxir(uulv"")
Σラ Lgg
In this case, gx(q) := L. Σタ= bgg for s 1,... .......,......... m. Since | = m is finite, we know that there exists a constant C1, 02 > such that e I 1j I C2 for 1 S S m, which implies that max,e(m] I|os (Ac. I G for some constant G based on k.
Moreover, estimating (q) based on the observations of X = (X ... .x)) and y is sufficient generate samples from the posterior distribution Pxjy (u,ly)) for 1,..- m. Since the exact form of ys is known, if we let Px- (Ir = u,ly y²) = (n,,(ss((T/n))) where T/n denotes the empirical of the training set, we obtain the following theorem.
Theorem 3. Under Assumption for any k N+, is [2
is finite, then there exists an approximate posterior Px (xlyy- = y²) satisfies that for any s (1,..., ,
Ex Pxx = u. = 5²)} Pxxr (ualy") Ok.m, ")
Varx Px- u.ly = y¹)} = Ok.m,o(n
The proposed sampling scheme in Algorithm] [ [generates k samples and linear combination of whose distributions approximates the posterior. In applications where is desired still generate one sample (rather than using a linear combination), we may consider a rejection sampling algorithm based on Theorem to sample from Px" (xly v²). Let T (Ti,-- .Tm) where T, -' 1 Lx,=u; for Then (Tr(T/)) (T)n), ... 9m (T/n)) is the posterior probability vector associated with
j == 1, the plug-in posterior Px (æly = y²) and probability vector associated with the debiased posterior Px- (xly y²) The rejection sampling described in Algorithm 2
... D..,k (T/n)) is the posterior

---

Algorithm 2 Rejection Sampling for Debiased Posterior Px = y²)
Input: Plug-in posterior (xn(x = =y"), debiased posterior Pxn (x y y²), large enough
constant M >> O
Output: Sample from the dehiased posterior Px- (x y=y y²"
repeat
2: Sample x' Px. ( (I y = y²)
3: Sample u ~ Uniform M)
4: until u Px=(1'
v²)
Px (x' 1y y*)
5: return
In Algorithm 2
Px (zly= v-)
D Dr,* (gx (T/m) g, (/n))
M max #EX Px- (zly=
max
is the upper bound of the ratio of the debiased posterior to the plug-in posterior.
5 Experiments
In this section, we provide numerical experiments to illustrate the debiasing framework for posterior approximation under the binary prior case and the Gaussian mixture prior case.
Binary prior case. Suppose that X' {0. 1} and X ~ Bern(g) for some unknown prior g E (0, 1).
Let a = oly") = Prix (y" (1) Pyxx (y" (0) be the likelihood ratio. Then the posterior distribution is give by X|Y ~ Bern( (gg/ (aq + 1 4)). We estimate g(g) == aq/ (aq + 1 - -q) based on the observations of X" and y"
Proposition
1 provides a debiased approximation as special case of Theorem 2 when | = 2.
Proposition 1. Let = >i X,, For k 1,2,3,4 4, we have
Ex {D,, A9 (T/n)] ((4) O(n On-
where Dr.k = E BA, and (a(g)()) = En- a6 (x) (1-2)) is the Bernstein
polynomial approximation of
In the proof of Theorem we notice that for any & E N+ Ex- (Dr,g(TT/n)) = which allows Proposition to be verified in closed form. To validate this result numerically, we consider two parameter settings: in the first experiment we set 0.4, y" 2, and Y|X ~ N(X, 1), while in the second we set q 3/11, y" 1, and Y|X (XX
For both settings, we examine the convergence rate of the debiased estimators D (T/n)( for == 1, 3, 4. The results shown in log-log plots in Figure where the vertical axis represents the logarithm of the absolute error and the horizontal axis represents the logarithm of the sample size n. and -4 are included for comparison.
Reference lines with slopes corresponding to n n-1 n.nn"
Gaussian mixture prior case. Suppose that X ~ NO, 1) (1, 1) and Y X + where { &~ (O, 1/16). Additionally, let y 0.8 and A {o 0.5}. In this case, validate the theoretical convergence rate
Exn
0
Since Ex- (++))(((*(())) does not have closed-form expression, we approximate it using Monte Carlo simulation. To ensure that the Monte Carlo error is negligible compared to the bias O(n -k), we select the number of Monte Carlo samples N such that >> n 2h - In practice, we run simulations for l = 1 and & 2 and set N n² for k = 1 and N n for l: = 2.
The results are shown in Figure 2 The figure presents log-log plots where the vertical axis represents the logarithm of the absolute error or of the variance and the horizontal axis represents the logarithm of the sample size n. For both and 2, the observed convergence rates align closely with the theoretical predictions.

---

20.4
10-4
10, cofhe
10871 seuarca
36g scube
(a) q = 0.4,y" = X ~ (X)
(b) = 301=3/11,yy - N(X
Figure I: Convergence of plug-in and debiased estimators in the binary prior case. The plot compares the approximation error of (TTxg(T/n) (k 2, 3, 4) against n. Reference lines with slopes corresponding to n n n² n and n are included to highlight the convergence rates.
Winson hog XcoM
1op Xccee
10
(a) Bias convergence rate
(b) Variance convergence rate
Figure 2: Convergence of debiased estimators in the Gaussian mixture prior case with ~ 1/061/16), y" = 0.8, and A {z 2 0.5). (a)
[N 1) + 1), Y X + + shows the bias decay of Dn,af((*) for l = 1, with reference lines slopes corresponding to n n1 and n -2 included for comparison. (b) shows the corresponding variance decay, alongside reference slope corresponding to
Conclusion
We introduced general framework for constructing debiased posterior approximation through observed samples D and the known likelihood Pyjx when the prior distribution is unknown. Here, naive strategy that directly plugs the empirical distribution into the Bayes formula or generative model has bias, because the likelihood is nonconstant, inducing distribution shift, and the map from the prior to posterior is nonlinear. It can be shown that the plug-in approach generates X with bias ||E,((P .D) Px|y || O(n -') and variance (P x| .D) O(n ('). In contrast, proposed debiasing framework achieves arbitrarily high-order bias rate of Onn for any integer k while maintaining the order of magnitude the variance. Our framework is black-box in the sense that only need to resample the training data and feed it into given black- box conditional generative model. In particular, we provide a rigorous proof for the Bayes-optima sampler under the continuous prior setting and for broad class of samplers with continuous 2k-th derivative, as specified in Assumption| 2 under the discrete prior setting. We expect that the proposed debiasing framework could work for general f and will support future developments in bias-corrected posterior estimation and conditional sampling.
10

---

Acknowledgments
This research was supported in part by NSF Grant DMS-2515510.
References
[1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2023.
[2] Kevin Black, Michael Janner, Yilun Du, 1lya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
[3] T. Tony Cai and Mark G. Low. Testing composite hypotheses, hermite polynomials and optimal estimation of a nonsmooth functional. The Annals of Statisticx, 39(2):011--1041 2011.
[4] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In USENIX Security Symposium, pages 5253-5270, 2023.
[5] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh Interna- tional Conference on Learning Representations, 2023.
[6] Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 1 117/49):30055--006, 2020.
[7] Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. In Advances in Neural Information Processing Systems, 2023.
[8] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9902-99155 2022.
[9] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances Neural Information Processing Systems, 2022.
[10] Hyun Lec. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges Representation Learning, ICML, 2013.
[11] Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479:47-59, 2022.
[12] Qiang Liu and Jason Lee. Black-box Importance Sampling. Proceedings of the 20th International Conference Artificial Intelligence and Statistics, volume 54, pages 952 2619, 2017.
[13] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the 39th International Conference on Machine Learning. 2022.
[14] Sebastian Nowozin. Debiasing evidence approximations: On importance srppotancee-eiighted autoencoders and jackknife variational inference. In International Conference on Learning Representations, 2018.
[15] Chris J. Oates, Mark Girolami, and Nicolas Chopin. Control Functionals for Monte Carlo Integration. Journal of the Royal Statistical Society Series B: Statistical Methodology, 79(3): 695-718, 05 2016.
11

---

[16] Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning, pages 26517-26582. 2023.
[17] Maurice Henry Quenouille. Notes on bias in estimation. Biometrika, 43(3-4):353-66)) 1956.
[18] Louis Sharrock, Jack Simons, Song Liu, and Mark Beaumont. Sequential neural score estima- tion: Likelihood-free inference with conditional score based diffusion models. In Proceedings of the 41st International Conference Machine Learning, volume 235, pages 44602,
2024.
[19] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[20] Kaizheng Wang. Pseudo-labeling for kernel ridge regression under covariate shift. arXiv preprint arXiv:2302. 10160, 2024.
[21] Qixun Wang. Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, and Yao Hu. Instantid: Zero-shot identity-preeee presservng generation in seconds. arXiv preprint arX.192401.07519, 2024.
[22] Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical bayes smoothing. In The Thirty Seventh Annual Conference on Learning Theory, pages 4958- 4991, 2024.
[23] Yihong Wu and Pengkun Yang. Polynomial methods in statistical inference: Theory and practice. Foundations and Trends@ in Communications and Information Theory, 17(4):402-586, 2020.
[24] Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal 1lms. In Forty-first International Conference on Machine Learning, 2024.
[25] TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K. Ryu. Diffusion probabilistic models generalize when they fail memorize. In ICML 2023 Workshop Structured Probabilistic Inferecce Generative Modeling, 2023.
[26] Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang. Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. In Advances in Neural Information Processing Systems, volume 36, pages 60599-60635, 2023.
[27] Andrew Zammit-Mangion Matthew Sainsbury-Dale, and Raphaël Huser. Neural methods for amortized inference. Annual Review of Statistics and Its Application, 12, 2024.
[28] Kaihong Zhang, Heqi Yin, Feng Liang, and Jingbo Liu. Minimax optimality of score-based diffusion models: Beyond the density lower bound assumptions. In International Conference on Machine Learning. 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The model setting and the assumptions our claim are clearly stated in abstract and introduction. Our main contributions stated in introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide a black-box debiaisng framework since we only provide the rigorous proof for the Bayes-optimal sampler under continuous prior setting and broad class of samplers with continuous 2k-th derivative under discrete prior setting. We expect the framework could work for general samplers f.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions are clearly stated or referenced in the statement of our theorems and lemmas. proofs appear the main paper and the supplemental material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We describe our experiments in the Experiment Section in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
14

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We used the simulation data the experiment.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://mips.co
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips. htcp//blcc/guddeeeeeeeeeieeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the setting of our simulation in the Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification:
Our experiments do not include any error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: Our experiment a simple simulation for binary case.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp::///eurpp::::: http://aawarr...ccoaaaaaaccccaaa
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our main contribution is constructing debiased approximation of the posterior distribution which does not have immediate societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy hate:p///eerrippc.cc////oeeerers for what should or should not be described.
19

---

A Proof of Theorem 1
Proof of Theorem
We begin by introducing notations that facilitates the analysis. Define
MA S {(z))(dx)
{(z)<(dz),
p
x
Let (0) = ,, x1) = के and set (x x = (X1, Xn). For j = ,,...,k k, we define x(0)
as the empirical measure of i.i.d. samples (x (5-1 (j-1) Furthermore, for each j = O,..., k, define
X 1) drawn from the measure -(i-1),
n
e
Σ ((x(0)) u4 u n n-1 Σ
((x!))) x(()
Let
k
kk
C
Σ (-1)--1 111 B
f=1
that it suffices to show that
(AA = On,
since Dr,k = Cn,k-
The Radon-Nikodym derivative of f(r with respect is
df(m) (T)
f(x:)
(x)
dn
J ((x))(dx)
For the empirical measure ît, the corresponding Radon-Nikodyn derivative of f(i) with respect to î takes the form
((x)
i E { (1) 1 x!!0)
(+t
(x)
(dr)((dr)
di
x
0,
otherwise,
((x)
if x x
Ei ((x(0)))
0,
otherwise.
Consequently,
= =a.
E { df( (z))
di
((x)
E
Σx_ (x) a(dx)
D. {(X) (0) )6 (xx(0) (A)
E
Σ ((X (0))
Moreover, by the definition B, and iterated conditioning, we have E (f(())(())) = E [ (((+())()))--01) E ( (B,f(((0-))(A))
E f(*())
(A).
20

---

By the same logic, for any j = 2....,k, k, we have
n Ei=l f(X, ((x(111)) 10 xx (A)
E [(( =E
n- E 1)
Thus,
(n) = Σ (1) (-19- (+)
j=1
(-18- E { {(+)))(AA}
=
n2 Σ i== ((x,1- 10 10x x(0--) (A)
Σ
11
n Ei= ((x)
MA (1-1)
= Σ
1-1
(1-1)) + 11
En
Then 9
holds if
1)
Σ 1-1)-11 111--BB
E
PA
PA
(10)
د +
en
11
1
Now we show that (0)) holds. By using the Taylor expansion of 1/(e/ (5-1)) + n), we have
2k-
(-1)"
(en 11
+
Σ
(e
(2
+
1-1)
1+1
g2k+1
6n + 1
(1-1) 1) + μ and μ.
where & lies between en
Since min(eli ( + n4 2 L1, we have 1/522+11 < L 2h-1 Additionally, L1 S ((x((-11) s L2
and Hoeffding's inequality implies that
212
P([ne [)
nl(T2 I,
for all t > O, which is equivalent to
2nt²
P(leli-211
S 2exp
(T22 - 12
Then
E
(1)2-
x
P
(j-1)
> t dl.
n
0
p
(111)
> 1/2k dt
2reu
<
2ku k-1 exp
du
(Inz -T1
-2kn -k 1 0 exp
2v
[cxp]
1 do
(I2- - -1,
= -o
21

---

Therefore, we have
2k (-1
PA
+
E
p
(e 1)0 }
+ O dn R),
+
p1
which implies that the I.H.S. of 10 can be written as
UA
+
Σ
(08-
+ +mn,agaan
Thus to prove the bound on the bias, it remains to show that for any given & and r 2k 1,
BRG
Σ j=1
k (-1-- B [u (el = On.Laann
Define a new operator B h [+{((F) for any h Ax + R and let
h₂(TT) = U (w))) } { ((o)m(as)))
Since BTh,(T)- E (ha(t)))) we have
k
Bg =
Σ
r
B' (T) (116001---
5=1
==0
Σ
We claim that Ber = Ol, Lo (n )) holds if for any 0 S s S Sk -
(I - OL.
(11)
Indeed, (T) O L2 is equivalent to ()= (-1)--1Bhh,, h,(#))
Onntos,nn- Therefore
11I implies
BKr
OL, (n)
Σ
(-1) "AP + (1
= (n
Now, prove the bound on the bias we only need show that 11 holds. For any k E and s E N+ let
J.
(a)s,u): a (a1, (21,a2,..)) ..,. (81, .), a;, E N+ N, 22 2 Σ
0;8; + v =8
and
Unso SEewrtaaa II /
(c) {
Jaasol S Cn (s)n
(a,8.v)E3.
Since ha( E Ad Lemma implies that (T- B) (T) A Therefore, (T - h) Or (n -k), finishing the proof for the bias bound.
where Co(s). (s),... are constants from Lemma
22

---

Finally, to prove the bound on the variance. consider the function F(z. v) = z/v. By construction,
f(* = F (-1) -el-11) + 1t).
Applying the Taylor expansion of F(x, y) yields
(()=)) +
MA (u
1 6-1) UA))
MA (1-1) 122
1 (-11) - MA)e + Ex c (
(1
for some Er lying between μ* and / ---" and Ey lying between 1a and e + μ. Since I s
(j-1) , en (111) + l S Lg implies that | [1/5]] and (5./5/1 are bounded by some constant
depending on L1 and L2. PA.IA
Moreover, since
L {(uX-1) -MA)e
= Cov (uy- "0li-),eli-11 ,eli-1) + +11)
< Var ( 11 1/2
11} 1/2
Var (15-1) +
1/2
}
1/2
Var n
Σ
1(x (1)) ((x x
(()-1) (4)
Var n1
E
((xx--1)
i=1
Var { (e(x(()) xip- (A)) 112
1/2
1/2
Var (e(x((-)))
n
= (n
and
E c 1) = Var (e(x!(-1) On..aan
Combining these bounds with the Taylor expansion, we conclude that for any 2 1,
(L)((T( E (A) ((T) (()) + OL,,2,n
By the same logic, we also have Bn ({()) = [ + (n Therefore.
( k
Da, (T)
= Σ
== ++11 (-1))A))()(A))
Σ (+41)-1)) (S(T)(A)) + O
==
(A) + +LL.tsh(1
and
= E -t..a//HAla--.////////////------ - -(+D)
Varx-
= -Bn (
= -Bn (A) + O -11) -
-A)(T)(A) +
(n
-Bn (A() + +L,,La (A)}
23

---

Lemma 2. There exist constants Co(s), Ci(s). C2(s),.... ..... such that the following holds.
For any K N and 8, n N+, les
3. = (a) (a1, ag........................ .2....... .2,...), N U.e.NN > 02 >...2 2...... 2 Σ
848; a,s
and
E a {1 ((()))(d)))
II { ea (2)
A
|aas,vl S C2(sm
(a,8.v)E3.
If h((()) E A2. then for any lo E N, we have
(I- h(=) E A
(12)
where B is an operator defined as Bh(x) B{h(a)} where ă the empirical distribution of
X1, X2...., X Lid. n
Proof of Lemma
We begin by proving that (T - B))(T) E A!. Since h(T)) E A.. let
Σ On,5, (1 (())m((d)))
II IV e (()x))))))
M(T)=
(n,
Note that [3.| does not depend on n and Jaas,ol S Co(s). It suffices to verify that each individual term in the sum satisfies
II { (t (x) E (1 B) B) [ (()))(()))
Without loss of generality, let (aI,... Ap) and (s1....,3p). s'= Sp==
Then we have
D 0;8; + v== s and
B B U (won(a))
II { e (())
R e- (=))
P
en (z))
E
1
n' E
e" (xx))xxx(A) (A)
ea (X,,)
For the term 1
j= 80 (X,)
let m
(8) denote the times X3 appears with powers a,, then
we have 23 m
3, for 1 i p. Define
I
m (m)²) jE(m).e(())
Σ
ml (0) m = s, for i E lp]
Therefore,
81 I1 - em (X,)
(xxx)xxx
e (x (x))xxx(1)
Ca.m
Di a:m (6)
(Xz),
mET.
==1
(13)
where
p
8
Ca.
(1)
m
24

---

expand Note that cam does not depend on n. Now we expand R.H.S. of (13) based on the number of distinct
variables X, appear, i.e. 2j=1 Eis amm)"> >0' which is equal to د}= E my n0 Define
m (
Im
0 o
then we have 1 5 Jml S 8'. Hence,
E
ee (xx))xx, (A)
eai (X²)
ed (x (A)
Σ
Σ
cam II
e a:my (X²)
m=
m
13 =m
n
(X,)xxxx (A)
II
Dim a.m (Xx)
Ca.m
m=1
mEZ [Jal mm
1=1
nn 1)... .. 8') cs)cs.m {1A (
P
[E (ea: (X)))*
12 II
+E
Cz.m
Σ (Xx))xx, (A)
E amm (Xx)
mEL. Imml=8'
BEJm
f=1
+
Ca.m
(X)))xx (A) II Ci apmp (Xg)
(14)
meI,
IJm -=
where Co.m
I 8t!.
The three terms in 14 are interpreted ( as the sum of many product terms of the
follows:
EEE (x())xxx, (A)) IT form ('())) [T= I par (Xj.)) The first term in 14 corresponds to the partial sum of terms in which all of X,, (Xj.)i. are distinct. The second term in (4] corresponds to the partial sum of terms in which X, is identical to one of (Xi,,)i; )i. while the latter are distinct. The third term corresponds to the partial sum terms in which at least two of (X,,.)s.t are identical. The last two term in (44 are at least O(n factor smaller than the first (due to fewer terms involved in the sum because of the constraint of having identical terms), while the first term will cancel with I h(x) when applying I - B to h.
denote the set of all distinct permutations of the vector consisting of non- zero elements .,,... bb bm and n - m zeros. Note that even the values of b, may be the same, still treat the b,s distinguishable. Then since Os are identical, we have (P(b.....,bml) nn - 1) ... (n- m + 1) = (nn)). Additionally, for any a and m, we define
Let P(bi,...,.
V (a, m)
Σ
aami
Σ amm!
1=1
Now we can write 14) as
nnn - 1) ... (n 50) c')c,, { ((()))(())
P
II [X]]]"
25

---

172
B [1 / pe (1))(())
E
E
Ca
by: b:: bi=s-U m.W.am))
itte
m2
Ca.m E
(X,) (4)
II
e (Xa)
by Di bE=s-o m.
=n(n- 1). (n s sa)cam U ((())((a))
II [E (x))1*
Σ
On Jes.m Σ
HE(B
II
[B[
/ (o)
by Eis bL=8-2
ift
Σ
O(n cs
('(x)xxx [1 (xx
b bE=s-v
Therefore,
(I (T- B) [V (()))(())) II
e (o)
n s' n(n 1) (n 8) cam
((())((d))
II 1m (X))I (X))*
n
O(n c
II BBI
/ (ott
by: by bu=
t=1
m O(n mmn (xx))xxx (A)
II c (x,))
by: Eb=: ma-
m
n s'+1 n(n - 1) ... (n 8)
Ca.m U (()))(d)))
II [IBE
n1++1
Σ
Oln )amm
II B (2
/ (
be bE=s-v
mm
Σ [[( / (
m
Σ
O(n m
Cs.m
be: by =8-F
"Cmm- {A (()))())))) }
Om
II [x)))
m=l
be b
by m
A
The last inleusion follows from that fact that (n²' nnn 1)... s'))/nn'+1 =OOn'' and the number of solutions to E by = does not depend on but depends on set
Now suppose (12) holds for k. Then
d'sso {A (()))(a))) ) II { (0)))))())
(T - h)))())=
Σ (a.s.)EJ.
26

---

where d S Cn * Then for k + 1, we have
da B) { (()))(d))) II / e (()))aaa)))
}
(I h(TT)
Σ
(a.9,0)E3.
Since for all a, S, such that v) 3s. (L2 (dx)) II (f (x) r(dx)} E Ag, we
have (T (T-B) B) {Sa e" (x)m<dx)) II. {S (2)()) x(dx)) A namely,
(T T B) { (()))(da)
IV (2)()) a((o))
Σ (b,t,((,,s,)) {AA (())m(a)))) II (0)
(b,t,a)e3.
where (ab,t,u(a,55 S, S Co(s)n Therefore.
(I - (I-B)h+ 13)
{ (())((()) II { ( (0)))(())) 4
Σ das.o
Σ
abee (a...u(a.ss))
(a,
(b,t,))3.
Akt1
B Proof of Theorem 2
In order to prove Theorem 2 first make some preliminary observations.
Let function f defined on the simplex Am = {q 1 generalized Bernstein basis polynomials of degree n as
9j 2 0,
93 = =}. Define the Eド=
(q) 9"
11
q"
Lemma 3. EEve (v/n bn (1)) S 1n -lall1//
Proof of Lemma 3 It sufficies to show that Eves (v bn. Z n|la/ll//, Since q Am. we treat Tn, =\t Eves,, (v (g) (q) as a function of the variables yı, 9mn---
For any 3 E Nm-1 such that ||/||] = 1, we let 7 = 7(3) = (eT, , Additionally, let
(0, .0. 0) E N Since
g0 - (vv=) -n nq) nq)
na (v - nq)a-o
and
88 (q) = n1)
(m) qu - 20 0
(
= =b) ( 2 (v- nq)² qa (v - nn)) }
we have
8TTna Σ
(
Σ (v- nq) gr
+EAm.
VEÂ...
1 - go Tn,00
= =-no"T + nox 0T Tn.a- + q"
Taaty
i.c.,
q"
q70ST, =nna"qq1 m + na
T m + Tnaty
go
T
n
27

---

By summing the above equation over 3 E N such that ||
1, we have
Σ
T0 n.
1=
1- qo
=-n Σ
a nna-- na
Σ
T
Σ
Tnatt
Tn, Tn,a+0
q0
1
|||1=1
||@||==1
1 - 9
Σ
qT n + naa - +
Σ
Tn,tt7
T n,++
q
=|
||1|===
n Σ a²q 1|||==
n1 T q Tn.a+0:
1
where the last equality follows from the fact that -1 Inaat + Tn 0. Therefore, have the following recurrence formula:
Tn,a+0 = -nq Σ
q a + na n - - -
Σ
qT00Taa (15)
1
||
9" T no q na TT + 8
Tn,ay = q0
(16)
Using the recurrence fomrula (5). T6) and the fact that T (1.00..---000)) .0TT = 0, (m.00.0...00 = nqn
91), (...........000000 -nn122 T,a the following form by using induction:
|||x]|///1
Tna = Σ
n
Σ 9",
(17)
j=1
n
where Cjn is independent of n. Then we can conclude that an S n nl/lax11/2) S n/l//1/2
Proof of Lemma 7 We prove the theorem by induction on k. For 1, by Taylor's expansion, there exists & E Am such that
)( = f(a)= Σ a f(E) v a
n
n
a=
Then we have
(Bx()) - = Σ
{ - f(a)) (g)
n1
vEăm
Σ ga ga f(E)( v4
agaa
(
12
PEAm
||a|||=1
|Iflc
Σ
-
q)a a (g)
n
rell==
1/2
< (fllct))))
Σ
Σ
v
q
2 bn. (q)
||0||1=1
DE
$ [[loo(4m.
Σ
n1 112
||a||1=11
S. 1lffloc
28

---

where the second inequality follows from Saachy--Scwwrzznneee inequality, and the third inequality follows from Lemma 3
Suppose the theorem holds up through k. Now we prove the theorem for l: + 1. For k++ 1, by Taylor's expansion, there exists & E Am such that
0 f())
(0 V
f( )( - f(q) + 7
Σ
( -q)at
Σ
a! (
q)aa
ad n
n
||||| m1
||a|
Then we have
(x(l)(q) - f(()==
E
- (q)
n
mEdm
k
8 f())))
8 f9((E) v 9)a (nul())
Σ
( q)aa g++
an!
||+|1==+11 ad!
vEAm
|||||1==
Σ
ja ((a))
Σ "" uEAm
v
)) ba)))
ad
|=/==1
da fCE(
Σ - q) veam
(())))) c00
+
ad!
|
Therefore,
(Br, ((())))))))1()
Σ |||||==]
(Bn 1) [)((411)/21-1
8 f(a)
Σ - 9) -
ad!
MEAm
Σ (Bn 1) ||0|1=441
ga f(E)
Σ - - (i-))annud bn
od!
v
(18)
We know that (a!)-taa Eve (v/n- b))) E Ch+1-|lalby (Am) since f E Ck+1 (Am). By the induction hypothesis, we have
First, we consider the first term of the right-hand side of 18
f(q ad!
(Bn 11
v Σ
q)) ())
n
DEam
80 f(q)
v Σ
-
q) a
a!!
n
Let
j ((q)
9a
Σ - q) (q))
ad
vEÂm
For any [.3| < ls +
||a|| we have
1108 la (9)lo =
Σ
3 poattslq)aa
Σ
-- q) a)mll
a!!
n
OSYSB
AEAm
Skt1 [1[(
Σ
28-2 y
V - - b
7
n
05
vEá...
29

---

Sa+1 Ilflllottssmm -lall///
where the last inequality follows from the fact that || n 2 which can be derived by using the form of a in 17 Therefore, we have
|00 y {
EEessm (v/n 9) (n)lll 00 2
0 flq)) (Bn 11)(0tt1-1aa1)/11
Σ
- q) b))
ad!
Se-1ll/llcars(aa..) (k+1)/2
Then we consider the second term of the right-hand side of (88)
2 F(E)
Σ
(Bn - ())
Σ A
V
q
ad!
n
|||||==411
SAtIll(nn - ( -1t
Σ
q)aº (n
n2
||||| =k+1 vEAm
1/2
S*+Ill(Bn - -11
Σ
Σ
- bn (q)
q) 2 9)
n2
|a|l==
wEAm
- - 11000- (k+11//2 (K++))/2
Finally, we have
ja sq)
[BB
<
Σ
(Bn, - 1) [)](+1))//)--11
Σ
q) 9) - bn
a!
|a
vEAm
8a (()
Σ
(Bn - 1)
Σ 1 9) )))) bn
ad!
|a||==kt1
vEAm
Σ
I( 1) fallllllltttt))) (k+13/2
Sk+1m
|||||==1
+ ||(B T) | -k++1)))
(k+1)/2
Sx11mml1fflccm1111)... 12-11)/2 -(t1)/
The last inequality holds when |B 71150 is bounded by constant independent of n. In fact, [|,
I)f loc = supqe4m. (BB, fg)( = supqEAm (/le//)) - f(q))b) (q) < 2
and B,, - 11o00 = (E 51 [I 1)/lloc S (1)) /| < 2. O
Proof of Theorem 2 The first claim follows from Lemma I and the fact that
max.e(m] l1g. ca2 (Am) I G and Ex (D,, (T.T/)))))) = (o.)
Additionally, we have
Da,
k-1
Σ
i 1+1 1) ) (1
k
1=
Σ (5 k
k-1
i (+11 (-1)6 (9.
On.mao(n11))
J=
(q) + O O0,m,a(n11
30

---

and
Ex (DnnA(o)((Tm)))
Σ
vEAm
Bn (Dn,)( (q)
(D)) 2 + Ok,,,a(n=1)
{g-(g) (n Omm,o((n
(g) Ox.,a(nn
Therefore,
Varx (TT/)) = Exn (Dnnlo))(Tm)) - x (Dn, (o.)(
= (ā( + -kkm,o(n {ax(a) + (n ?
Ok.m,c(n
C Proof of Theorem 3
Proof of Theorem 3 By Theorem it suffices to let Pxn(x uoly vr) (T,i)o)(T/n).
Moreover, we have _ Da, (9. (T/n) =
31


Reviews:
[{"review_id": "Bw7GcfQQqf", "reviewer": "Reviewer_yaah", "summary": "This paper examines the problem of taking a sample of unconditional observations $X$ and a known data likelihood distribution $P_{Y | X} $ and estimating the posterior distribution $P(X | Y = y^\\ast)$. The approach in this work does not estimate a parametric function for the distribution of $X$ but instead estimates the posterior distribution using a weighted sum of empirical dirac-delta distributions obtained by an iterative data resampling process. The first main result is to present the form of the weighted sum of iterative resampling procedures and to prove that this approximates the true posterior distribution with an approximation error of $\\mathcal{O}(n ^{-k})$ where $n$ is the sample size and $k$ is the number of resampling iterations. Next, a similar theorem with simpler assumptions is proved in the case where $X$ takes on a finite number of discrete values. Two algorithms are presented: one for approximating the posterior distribution using a weighted sum of empirical distribution for $k$ rounds of iterative resampling, and one for drawing a single sample from the approximate posterior in the case where $X$ is discrete. An experimental section validates the theoretical claims in the case where $X$ is a Bernoulli distribution and $Y | X$ follows a Gaussian distribution where the mean depends on $X$ using $k = 2$ resamplings.", "questions": "* Can more experiments be added for more complex scenarios? How well does this method scale with dimension and distributional complexity?\n* How does the Gaussian/Bernoulli experiment unfold for $k > 2$? Is there eventually a limit to the benefit of increasing $k$? The theoretical results appear to show that increasing $k$ should provide increasing benefits, but I am skeptical that iterative resampling can really provide increasingly accurate estimates since no \"new\" information is really being generated.\n* Can the authors provide an intuitive explanation for the form of $D_{n, k}$? Even after reading through the proof of Theorem 1 a few times, it is difficult to understand why the form $D_{n, k}$ enables better approximation accuracy than the naive plug-in estimator.\n* Does this method overcome limitations of the empirical Dirac distribution in high dimensional settings?", "limitations": "Limitations and societal consequences are not discussed by the authors. Lack of discussion about societal consequences does not impact my view of the work. Lack of discussion about limitations does impact my view, in particular limitations that relate to the scalability of the proposed method in terms of distribution dimension and complexity. The method is only applied in a very simple scenario, which makes me think that these limitations might be significant.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* The goal of estimating posterior distributions using only unconditional samples and a known likelihood is an important and relevant problem. Providing a method for obtaining better approximations that the naive (and biased) plug-in empirical estimate that only requires straightforward iterative resampling and weighting plug-in estimates has the potential to be a useful and lightweight solution.\n* The level of technical theoretical details is high, and the paper proves several non-trivial results. The theoretical results appear valid from what I reviewed, but the details are very dense and time consuming to thoroughly validate.\n* The experimental results in the Bernoulli/Gaussian setting match very well with the theoretical predictions for $k=2$ resamplings.\n\n**Weaknesses**\n* The experimental section is extremely limited and only examines a single very simple 1D situation. Furthermore, only $k=2$ resamplings are studied, while the theoretical results appear to demonstrate significant benefit for increasingly large $k$. It would be helpful to see experiments on more complex scenarios and for higher $k$.\n* The method relies heavily on the empirical Dirac distribution. While this alleviates the difficulty of learning a parametric model, I would guess that it heavily limits the scalability of the proposed method. In particular, the method seems very susceptible to the curse of dimensionality and prohibitively large number of samples $n$ might be needed in high-dimensional cases. The paper mentions diffusion and other generative models a few times, but I am not sure whether the proposed method is really viable in those kinds of situations.", "was_revised": true, "final_justification": "The authors provided additional experiments using more complex distributions and provided additional exploration into the effect of $k$. I am not convinced that this method is practical for high-dimensional machine learning situations. Nonetheless, the statistical contribution of this paper seems solid. I weakly recommend accepting the paper.", "rebuttal": "**Q1:** Can more experiments be added for more complex scenarios? How well does this method scale with dimension and distributional complexity?\n\n**Answer.** Thank you for the thoughtful suggestion.\n    We have added an additional experiment to evaluate our method in a more complex scenario.\n    Specifically, we consider a setting where $X$ follows a Gaussian mixture distribution, that is, \n    $X\\sim \\frac{1}{2}\\mathcal{N}(0,1)+\\frac{1}{2}\\mathcal{N}(1,1)$, and $Y=X+\\xi$ where $\\xi\\sim\\mathcal{N}(0, 1/16)$.\n    Additionally, we let $y^*=0.8$ and $A=\\\\{x: x\\ge 0.5\\\\}$.\n    The additive Gaussian noise model was previously considered in the conditional generative model literature, e.g. ref. [22] in our paper.\n    We use this experiment to validate the theoretical rate $|\\mathbb{E}\\_{X^n}[D\\_{n,k}f(\\hat{\\pi})(A)]-f(\\pi)(A)|=\\mathcal{O}(n^{-k})$.\n\nFirst, we compute the ground truth value $f(\\pi)(A)\\approx0.8795$.\n    Next, since $\\mathbb{E}\\_{X^n}[D\\_{n,k}f(\\hat{\\pi})(A)]$ does not have a closed form, we approximate it using Monte Carlo simulation.\n    To ensure that the Monte Carlo error is negligible compared to the bias $\\mathcal{O}(n^{-k})$, we choose the number of Monte Carlo samples $M$ such that $M\\gg n^{2k-1}$.\n\nWe have run the code for $k=1$ and $2$. Specifically, we choose $M=n^3$ for $k=1$ and $M=n^4$ for $k=2$.\n\nThe experimental results for this setting are summarized in the following table, where each entry represents the logarithm of the absolute error between the Monte Carlo estimate and the true value.\n\n|       | n=10    | n=20    | n=30    | n=40    | n=50    | n=60    | n=70    | n=80    | n=90    | n=100   |\n|-------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| k=1   | -3.1113 | -4.1539 | -4.6486 | -4.9098 | -5.2122 | -5.3870 | -5.5418 | -5.6924 | -5.8139 | -5.9296 |\n| k=2   | -4.8037 | -5.8835 | -6.6880 | -7.1578 | -7.8573 | -8.0784 | -8.4541 | -8.7667 | -9.0330 | -9.2975 |\n\nWhen we plot these errors against $\\log(n)$, the resulting lines for $k=1$ and $k=2$ are nearly parallel to the lines of $\\mathcal{O}(n^{-1})$ and $\\mathcal{O}(n^{-2})$ respectively.\n    This experiment demonstrates that the proposed method scales well in the presence of distributional complexity.\n\nIn general, the curse of dimensionality may arise and depends on the specific distribution of $X$ and the likelihood function.\n    There is no universal relationship between $n$ and the dimension $d$.\n    However, to provide some intuition, we can give an example that illustrates how $n$ and $d$ may relate.\n\nSuppose that $Y=(Y(1),\\\\dots,Y(d))$ and $X=(X(1),\\\\dots, X(d))$ have i.i.d. components, and $L_1\\le p(Y(i)|X(i)) \\le L_2$ for $1\\le i\\le d$.\nThen we have $\\ell(X):=p(Y|X)\\in[L_1^d, L_2^d]$.\nConsider line 390 in our paper, the residual term can be rewritten explicitly as $C\\cdot \\big((L_2^d-L_1^d)^2/2L_1^{2d}\\big)^kk!\\mathcal{O}(n^{-k})$ where   $C>0$ does not depend on $k$ and $d$.\n    To ensure that this residual term is negligible compared to the final term retained in the Taylor expansion, it suffices to let $n$ and $d$ satisfy that $\\big((L_2^d-L_1^d)^2/L_1^{2d}\\big)^k\\ll n^{1/2}$.\n    Therefore, by choosing $n$ such that $(L_2/L_1)^d\\ll n^{1/4k}$, our method scales with dimension.\n\n**Q2:** How does the Gaussian/Bernoulli experiment unfold for $k>2$? Is there eventually a limit to the benefit of increasing $k$? The theoretical results appear to show that increasing $k$ should provide increasing benefits, but I am skeptical that iterative resampling can really provide increasingly accurate estimates since no \"new\" information is really being generated.\n\n**Answer.**  We also conducted additional experiments for Gaussian/Bernoulli setting when $k=3$ and $4$.\n    Under the binary prior setting, we have $\\mathbb{E}\\_{X^n}[B\\_n^s(g)(T/n)]=B\\_n^{s+1}(g)(q)$ for any $s\\ge 0$, which enables us to compute the exact value of $\\mathbb{E}\\_{X^n}[D\\_{n,k}(g)(T/n)]$ in closed form, that is, $\\mathbb{E}\\_{X^n}[D\\_{n,k}(g)(T/n)]=\\mathbb{E}\\_{X^n}[D\\_{n,k}(g)(T/n)]=\\mathbb{E}\\_{X^n}[\\sum\\_{j=0}^k\\binom{k}{j+1}(-1)^jB\\_n^j(g)(T/n)]=\\sum_{j=0}^k\\binom{k}{j+1}(-1)^jB\\_n^{j+1}(g)(q)$.\n    This allows us to directly assess the approximation error without relying on Monte Carlo simulation.\n\nThe experimental results for $k=3$ and $4$ are reported in the following table, where each entry represents the logarithm of the absolute error between the exact closed-form expression value and the true value.\n    \n|       | n=100   | n=300   | n=500   | n=700   | n=900   |\n|-------|---------|---------|---------|---------|---------|\n| k=3   | -13.1934 | -16.6413 | -18.2059 | -19.2293 | -19.9910 |\n| k=4   | -15.3086 | -19.6794 | -21.7267 | -23.0750 | -24.0818 |\n\nWhen we plot these errors against $\\log(n)$, the resulting lines for $k=3$ and $4$ are parallel to the lines of $\\mathcal{O}(n^{-3})$ and $\\mathcal{O}(n^{-4})$ respectively.\n\nThere is no limit to the benefit of increasing $k$ for fixed $P(X,Y)$ (if the dimension $d$ increases with $n$, then we may require $kd \\ll \\log n$ in the example in our answer to Q1). \n    Although there is no \"new\" information, the iterative resampling can still provide increasingly accurate approximation.\n    For example, prior work such as \"Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference\" by Nowozin (2018, ICML) used jackknife technique to reduce bias in estimating the log-marginal likelihood without \"new\" information.\n    Although both our approach and Nowozin's paper use linear combination to reduce the bias, our method is more computationally efficient. \n    Specifically, we only require resampling $k$ times, whereas the jackknife method involves a leave-$m$-strategy for $1\\le m \\le k$ and calculate the average within each $m$ which leads to substantially higher computational cost.\n\n**Q3:** Can the authors provide an intuitive explanation for the form of $D\\_{n,k}$? Even after reading through the proof of Theorem 1 a few times, it is difficult to understand why the form $D\\_{n,k}$ enables better approximation accuracy than the naive plug-in estimator.\n\n**Answer.** We have provided an intuitive explanation for the form of $D\\_{n,k}$ implicitly in Remark 2.\n    To summarize, we view the operator $B\\_n f(\\pi):=\\mathbb{E}[f(\\hat{\\pi})]$  as a good approximation of $f(\\pi)$, i.e., $B\\_n\\approx I$, where $I$ is the identity operator.\n    This implies that the error operator $E:=I-B$ is a \"small\" operator.\n    Under this heuristic, if $E f(\\pi)=\\mathcal{O}(n^{-1})$, \n    intuitively we have $E^k f(\\pi)=\\mathcal{O}(n^{-k})$.\n    Using the binomial expansion of $E^k=(I-B\\_n)^k$, we have $E^k f(\\pi)=f(\\pi)-\\sum\\_{j=1}^k\\binom{k}{j}(-1)^{j-1}B_n^jf(\\pi)=f(\\pi)-\\mathbb{E}[\\sum\\_{j=1}^k\\binom{k}{j}(-1)^{j-1}B\\_n^{j-1}f(\\hat{\\pi})]=f(\\pi)-\\mathbb{E}[D\\_{n,k}f(\\hat{\\pi})]$.\n    This representation motivates the specific form of $D\\_{n,k}$.\n\n\n**Q4:** Does this method overcome limitations of the empirical Dirac distribution in high dimensional settings?\n\n**Answer.** In our response to Question 1, we have explained how to avoid the curse of dimensionality and included a simple example to illustrate how the sample size $n$ and dimension $d$ may relate."}, {"review_id": "VlbUucQScR", "reviewer": "Reviewer_LUxm", "summary": "Let $(X,Y)$ be a joint distribution, with the likelihood $P_{Y|X}$ assumed known. The paper tackles the task of estimating the posterior $P_{X\\mid Y = y^*}$ from i.i.d. samples $X_1, \\dots, X_n$ drawn from the joint. It concentrates on the **bias** of the estimator, measured as the total-variation distance between the *average* of the estimators and the true posterior. The key technical idea is to apply an operator—borrowed from polynomial-approximation/Richardson-extrapolation ideas—to build an estimator whose bias shrinks as $O(n^{-k})$ while the variance stays at the usual $O(n^{-1})$, for any fixed constant $k$.", "questions": "See weakness section.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. As far as I can tell, the operator-based approach is elegant and may be applicable in a broader context. Although similar ideas might exist (or even be well-known) in approximation theory, their use in the current setting appears novel.  \n2. The results are non-trivial, and the technical proofs are carefully worked out. I did not spot any significant mistakes.\n\n**Weaknesses**\n\n1. My main reservation of the paper is its objective: eliminating the **bias** of the estimator is rather weak. For example, for a single sample $X$ from a distribution $p$ the empirical distribution $\\delta_X$ is completely unbiased. A more relevant metric is the **expected risk** that compares the actual estimate to the underlying truth before taking the outer expectation. Given the variance bound $O(n^{-1})$, the expected risk of the proposed estimator shows no improvement over the naïve plug-in estimator.  Reducing bias relative to the plug-in estimator is not as surprising as the authors suggest.\n2. The guarantees are essentially asymptotic—the constant hidden in the $O(\\cdot)$ notation depends on $k$. A quick check of the proofs suggests it can be as large as $2^k$, which further limits practical usefulness. Can the authors state the constant explicitly, or at least its order?  \n3. The assumption that the likelihood $\\ell(x)$ is bounded away from zero feels unnatural and needs justification.  \n4. In my opinion, this is a classical statistics paper packaged as a machine-learning contribution. I can hardly imagine how the claimed application scenarios (e.g., generative models) match the actual results. A concrete, convincing example would help.\n\n**Overall**\n\nDespite these criticisms, the paper solves a non-trivial problem in a niche area, and the work is solid. I therefore recommend borderline accept.", "was_revised": true, "final_justification": "Though I'm still skeptical about the practical significance of the work in machine learning, it is nevertheless a solid contribution. I therefore maintain my current rating, favoring the acceptance of the paper.", "rebuttal": "**Q1:**  My main reservation of the paper is its objective: eliminating the bias of the estimator is rather weak. For example, for a single sample $X$ from a distribution $p$ the empirical distribution $\\delta_X$ is completely unbiased. A more relevant metric is the expected risk that compares the actual estimate to the underlying truth before taking the outer expectation. Given the variance bound $\\mathcal{O}(n^{-1})$, the expected risk of the proposed estimator shows no improvement over the naive plug-in estimator. Reducing bias relative to the plug-in estimator is not as surprising as the authors suggest.\n\n**Answer.**  We appreciate the reviewer’s comment and agree that bias alone is not the only relevant criterion.\n    \nHowever, in many problems in statistics and machine learning, it is useful to reduce the bias, even if the variance remains the same order.\n    For example, \"Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference\" by Nowozin (2018, ICML) demonstrates the practical importance of bias reduction. \n    In his paper, the jackknife technique is used to reduce bias in estimating the log-marginal likelihood, even though it does not improve the variance rate. \n    Our contribution follows a similar motivation.\n    While the variance of the proposed estimator remains at order $\\mathcal{O}(n^{-1})$, we show that its bias decays faster than that of the standard plug-in estimator.\n\n\n**Q2:** The guarantees are essentially asymptotic—the constant hidden in the  $\\mathcal{O}(\\cdot)$ notation depends on $k$. A quick check of the proofs suggests it can be as large as $2^k$, which further limits practical usefulness. Can the authors state the constant explicitly, or at least its order?\n\n**Answer.**  The constant hidden in the bound $\\mathcal{O}\\_{L\\_1, L\\_2, k}(n^{-k})$ can be written explicitly in the form $C\\cdot \\big((L\\_2-L\\_1)^2/2L\\_1^2\\big)^k k!$ where   $C>0$ does not depend on $k$.\n    Although this hidden constant does limit the practical value for large $k$, the result is still meaningful for moderate values such as $k=2,3,4,5$ and helps provide theoretical insight into the asymptotic behavior.\n\n\n**Q3:** The assumption that the likelihood  $\\ell(x)$ is bounded away from zero feels unnatural and needs justification.\n\n**Answer.** We agree that the assumption that the likelihood $\\ell(x)$ is bounded away from zero may appear strong.\n    However, in many structured settings, such a lower bound can be justified with some more mild conditions.\n\nFor example, consider the Reward-Directed Conditional Diffusion model setup in Yuan et al. (2023, NeurIPS) that $y=f^\\*(x)+\\xi$ where $\\xi\\sim\\mathcal{N}(0,\\sigma^2)$ and the data sampling distribution $P\\_x$ is supported on a low-dimensional linear subspace, i.e., $x=Az$. They further assume that $z\\sim(0, \\Sigma)$ and $\\lambda_{\\min} I\\_d \\preceq \\Sigma \\preceq \\lambda\\_{\\max} I_d$.\n    If we further assume that the distribution of $x$ has compact support, we can ensure that the likelihood is bounded away from zero for any given $y^*$.\n\n**Q4:** In my opinion, this is a classical statistics paper packaged as a machine-learning contribution. I can hardly imagine how the claimed application scenarios (e.g., generative models) match the actual results. A concrete, convincing example would help.\n\n**Answer.** We have included a more sophisticated experiment in our response to Reviewer yaah's Question 1 and such additive Gaussian noise model setting was previously considered in the conditional generative model literature, e.g. ref. [22] in our paper. \n    Moreover, prior work \"Debiasing Evidence Approximations: On Importance-weighted Autoencoders and Jackknife Variational Inference\" by Nowozin (2018, ICML) has successfully applied jackknife technique to reduce the bias in estimating the log-marginal likelihood to real datasets like MNIST.\n    Their experiment suggests that for real data, there could be an intrinsic dimension much samller than the ambient dimension, so that the curse of dimensionlity may not be present."}, {"review_id": "N9CEj1GbXI", "reviewer": "Reviewer_FYyK", "summary": "This paper studies the task of sampling from the posterior distribution $P _ {X|Y=y^*}$ given i.i.d. samples $X _ {1:n}$ from an unknown prior distribution $\\pi$ and a known likelihood model $P _ {Y|X}$. The authors propose a novel approach to reduce the bias of the plug-in approximation (i.e., replacing $\\pi$ with the empirical distribution of the samples) so that the TV distance of the averaged posterior distribution (over the randomness of $X _ {1:n}$) to the true posterior distribution can decay at $O(n^{-k})$ for any integer $k$, while the average variance of the posterior distribution remains $O(n^{-1})$. The authors prove two cases of the continuous and discrete prior distributions, respectively, and validate their theoretical findings through numerical experiments on Bernoulli prior distribution.", "questions": "See weakness part above.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThis paper is theoretically solid and well-written. I have gone through the proofs and have not found any significant issues. The notations are mostly clear, though I had some difficulty understanding the definition of $B _ n$ and $B _ n^j$ at first read (but it was clarified in the proof and the pseudocodes). The mathematical derivations are rigorous, and the assumptions are clearly stated. The main results are also clearly presented, with an easy-to-follow sketch of proof in the main text providing good intuition. \n\n**Weaknesses**\n\nWritings can be improved in some places. For example, in typical machine learning conference papers, it is not very common to include heavy mathematical details in the abstract and introduction sections. For instance, content between lines 49 and 71 can be briefly summarized in the introduction but elaborated in the main body. The contribution part can also be summarized and itemized for better clarity. Also, the authors can convert some inline equations to display style for better readability (e.g., lines 385 to 389). I suggest the authors to refer to some well-written papers in top ML conferences for better writing styles.\n\nWhile I understand that this is a statistical theory paper, as the authors propose theoretically-grounded improved methodologies, the empirical results can be further strengthened, such as adding experiments on continuous prior distributions and trying estimators with bias $O(n^{-k})$ for some larger $k$. The rejection sampling proposed in Alg. 2 can also be tried. As the authors mentioned, the proposed framework can be applied to generative models like diffusion models and VAEs for conditional sampling, it would significantly enhance the impact of this work if the authors can demonstrate the effectiveness of their methods on some real-world datasets (i.e., with more complex prior distributions).", "was_revised": true, "final_justification": "The authors have addressed most of the concerns from me and the other reviewers during the rebuttal. While I believe the paper could still benefit from enhanced experimental results, the current form is enough for getting an acceptance, especially considering the theoretical contributions. I thus maintain my original evaluation.", "rebuttal": "**Q1:**  Writings can be improved in some places. \n\n**Answer.**  We appreciate the reviewer’s helpful suggestions on the writing.\n    We will revise the abstract and introduction by summarizing key ideas and deferring mathematical details to the main body.\n    We will also rewrite the contributions into a clear, itemized form for better clarity.\n    In addition, we have converted several inline equations to display style.\n\n\n**Q2:** Add experiments.\n\n**Answer.** We have added additional experiments to evaluate our method in a more complex scenario and also conducted additioanl experiments for Gaussian/Bernoulli setting when $k=3$ and $4$.\n    Please refer to our detailed response to Reviewer yaah’s Question 1 and 2.\n\n\n**Q3:** As the authors mentioned, the proposed framework can be applied to generative models like diffusion models and VAEs for conditional sampling, it would significantly enhance the impact of this work if the authors can demonstrate the effectiveness of their methods on some real-world datasets (i.e., with more complex prior distributions).\n\n**Answer.** We have included a more complex experiment in our response to Reviewer yaah's Question 1 and such additive Gaussian noise model setting was previously considered in the conditional generative model literature, e.g. ref. [22] in our paper.\n    Please refer to our response to Reviewer LUxm's Question 4."}]

Author rebuttal:

Meta-review: This paper presents a black-box $k$-th order debiasing framework that combines recursively sampled empirical priors with signed weights to yield an approximate posterior that achieves a bias rate of $\mathcal{O}(n^{-k})$ and variance $\mathcal{O}(n^{-1})$. 

Reviewers recognized the solid and elegant theoretical contributions as its primary strength, and the uncertainty around high-dimensional viability as the primary weakness. The rebuttal process was productive, with the authors adding new non-trivial experimental results and other mathematical details requested by reviewers, making the paper stronger. 

Overall, the paper makes a good contribution that, despite modest empirical results currently, can lead to more advances in the future. Therefore, I am happy to recommend acceptance. I strongly suggest that the authors follow through on all promises made to update the final paper during the discussion period.
Decision: Accept (poster)