Title: A Black-Box Debiasing Framework for Conditional Sampling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Conditional sampling

Bayesian Inference

Posterior Approximation

Bias Correction
Abstract: Conditional sampling is a fundamental task in Bayesian statistics and generative modeling. Consider the problem of sampling from the posterior distribution $P\_{X|Y=y^\*}$ for some observation $y^\*$, where the likelihood $P\_{Y|X}$ is known, and we are given $n$ i.i.d. samples $D=\\{X\_i\\}\_{i=1}^n$ drawn from an unknown prior distribution $\pi\_X$. Suppose that $f(\hat{\pi}\_{X^n})$ is the distribution of a posterior sample generated by an algorithm (e.g. a conditional generative model or the Bayes rule) when $\hat{\pi}\_{X^n}$ is the empirical distribution of the training data. Although averaging over the randomness of the training data $D$, we have $\mathbb{E}\_D\left(\hat{\pi}\_{X^n}\right)= \pi\_X$, we do not have $\mathbb{E}\_D\left\\{f(\hat{\pi}\_{X^n})\right\\}= f(\pi\_X)$ due to the nonlinearity of $f$, leading to a bias. In this paper we propose a black-box debiasing scheme that improves the accuracy of such a naive plug-in approach. For any integer $k$ and under boundedness of the likelihood and smoothness of $f$, we generate samples $\hat{X}^{(1)},\dots,\hat{X}^{(k)}$ and weights $w\_1,\dots,w\_k$ such that $\sum_{i=1}^kw_iP\_{\hat{X}^{(i)}}$ is a $k$-th order approximation of $f(\pi\_X)$, where the generation process treats $f$ as a black-box. Our generation process achieves higher accuracy when averaged over the randomness of the training data, without degrading the variance, which can be interpreted as improving memorization without compromising generalization in generative models.

A Black-Box Debiasing Framework for Conditional Sampling

Abstract

Conditional sampling is a fundamental task in Bayesian statistics and generative modeling. Consider the problem of sampling from the posterior distribution P X | Y = y ∗ for some observation y ∗ , where the likelihood P Y | X is known, and we are given n i.i.d. samples D = { X i } n i =1 drawn from an unknown prior distribution π X . Suppose that f (ˆ π X n ) is the distribution of a posterior sample generated by an algorithm (e.g. a conditional generative model or the Bayes rule) when ˆ π X n is the empirical distribution of the training data. Although averaging over the randomness of the training data D , we have E D (ˆ π X n ) = π X , we do not have E D { f (ˆ π X n ) } = f ( π X ) due to the nonlinearity of f , leading to a bias. In this paper we propose a black-box debiasing scheme that improves the accuracy of such a naive plug-in approach. For any integer k and under boundedness of the likelihood and smoothness of f , we generate samples ˆ X (1) , . . . , ˆ X ( k ) and weights w 1 , . . . , w k such that ∑ k i =1 w i P ˆ X ( i ) is a k -th order approximation of f ( π X ) , where the generation process treats f as a black-box. Our generation process achieves higher accuracy when averaged over the randomness of the training data, without degrading the variance, which can be interpreted as improving memorization without compromising generalization in generative models.

1 Introduction

Conditional sampling is a major task in Bayesian statistics and generative modeling. Given an observation y ∗ , the objective is to draw samples from the posterior distribution P X | Y = y ∗ , where the likelihood P Y | X is known but the prior distribution π X is unknown. Instead, we are provided with a dataset D = { X i } n i =1 consisting of n i.i.d. samples drawn from π X .

The setting is common in a wide range of applications, including inpainting and image deblurring [9, 5] (where X is an image and Y | X is a noisy linear transform), text-conditioned image generation [7, 13](where X is an image and Y is a natural language prompt), simulating biomedical structures with desired properties, and trajectory simulations for self-driving cars. Moreover, conditional sampling is equally vital in high-impact machine learning and Bayesian statistical methods, particularly under distribution shift, such as in transfer learning. For instance, conditional sampling has enabled diffusion models to generate trajectories under updated policies, achieving state-of-the-art performance in offline reinforcement learning [8, 1, 26]. Pseudo-labeling, a key technique for unsupervised pretraining [10] and transfer learning calibration [20], relies on generating conditional labels. Additionally, conditional diffusion models seamlessly integrate with likelihood-free inference [6, 18, 27]. Existing approaches often use generative models such as VAEs or Diffusion models to generate samples by learning P X | Y = y ∗ implicitly from the data.

Our work focuses on approximating the true posterior P X | Y = y ∗ using the observed samples D = X n = ( X 1 , . . . , X n ) and the new observation y ∗ , but without the knowledge of the prior. Denote by P ˆ X | Y = y ∗ ,D the approximating distribution. We can distinguish two kinds of approximations: First, P ˆ X | Y = y ∗ ,D ≈ P X | Y = y ∗ with high probability over D , which captures the generalization ability since the model must learn the distribution from the training samples. This criterion is commonly adopted in estimation theory and has also been examined in the convergence analysis of generative models [16, 28, 26, 22]. Second, E D ( P ˆ X | Y = y ∗ ,D ) ≈ P X | Y = y ∗ is a weaker condition since it only requires approximation when averaged over the randomness of the training data, but is still useful in some sampling and generative tasks, e.g. generating samples for bootstrapping or Monte Carlo estimates of function expectations. The second condition captures the ability to memorize or imitate training sample distribution. It is interesting to note that in the unconditional setting (i.e., without distribution shift), a permutation sampler can perfectly imitate the unknown training data distribution, even if n = 1 , so the problem is trivial from the sample complexity perspective. However, in the conditional setting, it is impossible to get such a perfect imitation with finite training data, as a simple binary distribution example in Section 3.2 illustrates. It naturally leads to the following question:

How fast can the posterior approximation converge to the true posterior as n →∞ , and is there a sampling scheme achieving this convergence rate?

Contribution. We address the question above by proposing a novel debiasing framework for posterior approximation. Our main contributions can be summarized as follows:

Debiasing framework for posterior approximation. We introduce a novel debiasing framework for posterior approximation with an unknown prior. Our method leverages the known likelihood P Y | X and the observed data to construct an improved approximate posterior ˜ P X n ( x | y ∗ ) with provably reduced bias. In particular, let f (ˆ π X n ) represent the distribution of a posterior sample generated by an algorithm f when ˆ π X n is the empirical distribution of the training data. Then for any integer k , assuming that the likelihood function P Y | X is bounded and f is sufficiently smooth, we generate samples ˆ X (1) , . . . , ˆ X ( k ) from f based on multiple resampled empirical distributions. These are then combined with designed (possibility negative) weights w 1 , . . . , w k to construct an approximate posterior:

which is a k -th order approximation of f ( π X ) , treating the generation process f as a blackbox. Our generation process achieves higher accuracy when averaged over the randomness of the training data, but not conditionally on the training data, which highlights the trade-off between memorization and generalization in generative models. Specifically, we do not assume any parametric form for the prior and our method can achieve a bias rate of O ( n -k ) for any prescribed integer k and a variance rate of O ( n -1 ) .

Theoretical bias and variance guarantees. We establish theoretical guarantees on both bias and variance for the Bayes-optimal sampler under continuous prior setting and for a broad class of samplers f with a continuous 2 k -th derivative, as specified in Assumption 2, under the discrete prior setting. The proposed debiasing framework can also be applied in a black-box manner (see Remark 2 for the intuition), making it applicable to a broad class of state-of-the-art conditional samplers, such as diffusion models and conditional V AE. Based on this perspective, we treat the generative model f as a black box that can output posterior samples given resampled empirical distributions. Applying f to multiple recursive resampled versions of the training data and combining the outputs with polynomial weights, we obtain a bias-corrected approximation of the posterior. The procedure is described in Algorithm 1.

Our approach is also related to importance sampling. Since the true posterior P X | Y is intractable to compute, we can use expectations under the debiased posterior ˜ P X n ( x | y ∗ ) to approximate the expectations under the true posterior P X | Y = y ∗ . For a test function h , we estimate E P X | Y = y ∗ { h ( X ) } by

Algorithm 1 Posterior Approximation via Debiasing Framework

Input: Observation y ∗ , likelihood P Y | X , data X n = ( X 1 , . . . , X n ) , number of steps k , a black-box conditional sampler f (i.e., a map from a prior distribution to a posterior distribution)

Output: ˆ X ( j ) , j = 1 , . . . , k such that ∑ k -1 j =0 ( k j +1 ) ( -1) j P ˆ X ( j +1) is a high-order approximation of the posterior P X | Y = y ∗

1: Initialize ˆ p (1) := ˆ π X n

2: for ℓ = 2 to k do

3: Generate n i.i.d. samples from ˆ p ( ℓ -1)

4: Let ˆ p ( ℓ ) be the empirical distribution of the sampled data

5: end for

6: for j = 1 to k do

7: Generate samples ˆ X ( j ) ∼ f (ˆ p ( j ) )

8: end for

9: Return ˆ X ( j ) , j = 1 , . . . , k

where ˜ X j ∼ q ( x | y ∗ ) for a chosen proposal distribution q . This resembles our method, in which we approximate the true posterior by a weighted combination ∑ k i =1 w i P ˆ X ( i ) . And in (1), the term ˜ P X n ( ˜ X j | y ∗ ) /q ( ˜ X j | y ∗ ) can be interpreted as a weight assigned to each sample, analogous to the weights w i in our framework. Therefore, we expect that Algorithm 1 can be broadly applied to Monte Carlo estimates of function expectations, similar to the standard importance sampling technique.

2 Related work

Jackknife Technique. Our work is related to the jackknife technique [17], a classical method for bias reduction in statistical estimation that linearly combines estimators computed on subsampled datasets. Specifically, the jackknife technique generates leave-one-out (or more generally, leaves -out where s ≥ 1 ) versions of an estimator, and then forms a weighted combination to cancel lower-order bias terms. Recently, Nowozin [14] applied the jackknife to the importance-weighted autoencoder (IWAE) bound ˆ L n , which estimates the marginal likelihood log π ( x ) using n samples. While ˆ L n is proven to be an estimator with bias of order O ( n -1 ) , the jackknife correction produces a new estimator with reduced bias of order O ( n -m ) . Our paper introduces a debiaisng framework based on the similar idea that using a linear combination of multiple approximations to approximate the posterior.

Conditional Generative Models. Conditional generative models have become influential and have been extensively studied for their ability to generate samples from the conditional data distribution P ( ·| y ) where y is the conditional information. This framework is widely applied in vision generation tasks such as text-to-image synthesis [13, 24, 2] where y is an input text prompt, and image inpainting [11, 21] where y corresponds to the known part of an image. We expect that our proposed debiasing framework could work for a broad class of conditional generative models to construct a high order approximation of the posterior P ( ·| y ) .

Memorization in Generative Models. The trade-off between memorization and generalization has been a focus of research in recent years. In problems where generating new structures or preserving privacy of training data is of high priority, generalization is preferred over memorization. For example, a study by Carlini et al. [4] demonstrates that diffusion models can unintentionally memorize specific images from their training data and reproduce them when generating new samples. To reduce the memorization of the training data, Somepalli et al. [19] applies randomization and augmentation techniques to the training image captions. Additionally, Yoon et al. [25] investigates the connection between generalization and memorization, proposing that these two aspects are mutually exclusive. Their experiments suggest that diffusion models are more likely to generalize when they fail to memorize the training data. On the other hand, memorizing and imitating the training data may be intentionally exploited, if the goal is Monte Carlo sampling for evaluations of expected values, or if the task does not involve privacy issues, e.g. image inpainting and reconstruction. In these applications, the ability to imitate or memorize the empirical distribution of the training data becomes essential, especially when generalization is unattainable due to the insufficient data. Our work focuses

on memorization phase and shows that it is possible to construct posterior approximations with provably reduced bias by exploiting the empirical distribution.

Mixture-Based Approximation of Target Distributions. Sampling from a mixture of distributions a 1 P X 1 + a 2 P X 2 + · · · + a k P X k to approximate a target distribution P ∗ is commonly used in Bayesian statistics, machine learning, and statistical physics, especially when individual samples or proposals are poor approximations, but their ensemble is accurate. Traditional importance sampling methods often rely on positive weights, but recent work has expanded the landscape to include more flexible and powerful strategies, including the use of signed weights and gradient information. For example, Oates et al. [15] uses importance sampling and control functional estimators to construct a linear combination of estimators with weights a k to form a variance-reduced estimator for an expectation under a target distribution P ∗ . Liu and Lee [12] select the weights a k by minimizing the empirical version of the kernelized Stein discrepancy (KSD), which often results in negative weights.

3 Problem setup and notation

Consider a dataset { X i } n i =1 consisting of n independent and identically distributed (i.i.d.) samples, where X i ∈ X is drawn from an unknown prior distribution π X and the conditional distribution P Y | X is assumed to be known. In the Bayesian framework, the posterior distribution of X given Y is given by

Given the observed data X n = ( X 1 , · · · , X n ) and the new observation y ∗ , our goal is to approximate the true posterior P X | Y = y ∗ .

3.1 Naive plug-in approximation

A natural approach is to replace the unknown prior π X with its empirical counterpart

in the Bayes' rule to compute an approximate posterior which yields the plug-in posterior

̸

Note that even though E D (ˆ π X n ) = π X , the nonlinearity of Bayes' rule makes the resulting posterior (2) still biased, that is, E D { ̂ P X | Y ( ·| y ∗ ) } = P X | Y ( ·| y ∗ ) . If the denominator in (2) were replaced with ∫ P Y | X ( y ∗ | x ) π X ( dx ) , then averaging the R.H.S. of (2) over the randomness in X n would yield the true posterior P X | Y ( dx | y ∗ ) = P Y | X ( y ∗ | x ) π X ( dx ) / ∫ P Y | X ( y ∗ | x ) π X ( dx ) exactly.

For typical choices of P Y | X which have nice conditional density (e.g., the additive Gaussian noise channel), ∫ P Y | X ( y ∗ | x )ˆ π X n ( dx ) converges at the rate of n -1 / 2 , by the central limit theorem. Consequently, E D ( ̂ P X | Y = y ∗ ) converges to the true posterior at the rate ˜ O ( n -1 / 2 ) in the ∞ -Renyi divergence metric regardless of the smoothness of π X . Under appropriate regularity conditions, we can in fact show that E D ( ̂ P X | Y = y ∗ ) converges at the rate of ˜ O ( n -1 ) , which comes from the variance term in the Taylor expansion. Naturally, we come to an essential question: can we eliminate the bias entirely? That is, is it possible that E D { ̂ P X | Y ( ·| y ∗ ) } = P X | Y ( ·| y ∗ ) ?

3.2 Impossibility of exact unbiasedness

Exact unbiasedness is, in general, unattainable. Consider the simple case where X is binary, that is, X ∼ Bern( q ) for some unknown parameter q ∈ (0 , 1) . Define the likelihood ratio α = α ( y ∗ ) :=

P Y | X ( y ∗ | 1) /P Y | X ( y ∗ | 0) . Then the true posterior is

On the other hand, if we approximate the posterior distribution as ˆ P X | Y (1 | y = y ∗ ) = Bern( p ( k )) upon seeing k outcomes equal to 1, then

which is a polynomial function of q , and hence cannot equal the rational function αq/ ( αq +1 -q ) for all q . This implies that an exact imitation, in the sense that E D { ̂ P X | Y ( ·| y ∗ ) } = P X | Y ( ·| y ∗ ) , ∀ π X , is impossible. However, since a rational function can be approximated arbitrarily well by polynomials, this does not rule out the possibility that a better sampler achieving convergence faster than, say, the ˜ O ( n -1 / 2 ) rate of the naive plug-in method. Indeed, in this paper we propose a black-box method that can achieve convergence rates as fast as O ( n -k ) for any fixed k > 0 .

3.3 Objective and notation

Since the bias in the plug-in approximation arises from the nonlinearity of Bayes' rule, we aim to investigate whether a faster convergence rate can be achieved. Our objective is to construct an approximation ˜ P X n ( x | y = y ∗ ) that improves the plug-in approximation by reducing the bias. Specifically, the debiased approximation satisfies the following condition:

More generally, we can replace the Bayes rule by an arbitrary map f from a prior to a posterior distribution (e.g. by a generative model), and the goal is a construct a debiased map ˜ f such that

Notation. Let δ x denote the Dirac measure, ∥ · ∥ TV denote the total variation norm. For any positive integer m , denote [ m ] = { 1 , . . . , m } as the set of all positive integers smaller than all equal to m . Write b n = O ( a n ) if b n /a n is bounded as n →∞ . Write b n = O s ( a n ) if b n /a n is bounded by C ( s ) as n →∞ for some constant C ( s ) that depends only on s . We use the notation a ≲ b to indicate that there exists a constant C > 0 such that a ≤ Cb . Similarly, a ≲ k b means that there exists a constant C ( k ) > 0 that depends only on k such that a ≤ C ( k ) b . Furthermore, for notational simplicity, we will use π to denote the true prior π X and ˆ π to denote the empirical prior ˆ π X n in the rest of the paper.

4 Main result

4.1 Debiased posterior approximation under continuous prior

Let ∆ X denote the space of probability measures on X . Define the likelihood function ℓ ( x ) := P Y | X ( y ∗ | x ) , which represents the probability of observing the data y ∗ given x . Let f : ∆ X → ∆ X be a map from the prior measure to the posterior measure, conditioned on the observed data y ∗ . Let B n be the operator such that for any function f : ∆ X → ∆ X ,

where ˆ p denotes the empirical measure of n i.i.d. samples from measure p .

We consider the case that f represents a mapping corresponding to the Bayes posterior distribution. Using Bayes' theorem, for any measure π ∈ ∆ X and any measurable set A ⊂ X , the posterior measure f ( π ) is expressed as

As discussed in Section 3, the equality B n f ( π ) = f ( π ) is not possible due to the nonlinearity of f . However, we can achieve substantial improvements over the plug-in method by using polynomial approximation techniques analogous to those from prior statistical work by Cai and Low [3] and Wu and Yang [23]. For k ≥ 1 , we define the operator D n,k as a linear combination of the iterated operators B j n for j = 0 , . . . , k -1 :

Assumption 1. The likelihood function ℓ is bounded, i.e., there exists 0 < L 1 ≤ L 2 such that L 1 ≤ ℓ ( x ) ≤ L 2 .

The following theorem provides a systematic method for constructing an approximation of f ( π ) with an approximation error of order O ( n -k ) for any desired integer k .

Theorem 1. Under Assumption 1, for any measurable set A ⊂ X and any k ∈ N + , we have

Remark 1. D n,k f (ˆ π ) = ∑ k -1 j =0 ( k j +1 ) ( -1) j B j n f (ˆ π ) in (5) can be interpreted as a weighted average of the distribution of some samples. Specifically, if we treat the coefficient ( k j +1 ) ( -1) j as the weight w j and B j n f (ˆ π ) as the distribution of some sample ˆ X ( j ) , then D n,k f (ˆ π ) = ∑ k -1 j =0 w j P ˆ X ( j ) .

Remark 2. Recall the binary case discussed in Section 3, (3) illustrates that we cannot get an exact approximation for the true posterior. But from (5) , we demonstrate that even if ∥ E X n { D n,k f (ˆ π ) } -f ( π ) ∥ TV = 0 is impossible, it can be arbitrarily small. Although the theoretical guarantees are derived for the Bayes-optimal sampler, (5) is expected to hold for general sampler f such as diffusion models. Here we give the intuition for this conjecture. We view the operator B n f ( π ) := E { f (ˆ π ) } as a good approximation of f ( π ) , i.e., B n ≈ I , where I is the identity operator. This implies that the error operator E := I -B n is a 'small" operator. Under this heuristic, if Ef ( π ) = O ( n -1 ) , intuitively we have E k f ( π ) = O ( n -k ) . Using the binomial expansion of E k = ( I -B n ) k , we have E k f ( π ) = f ( π ) -∑ k j =1 ( k j ) ( -1) j -1 B j n f ( π ) = f ( π ) -E { ∑ k j =1 ( k j ) ( -1) j -1 B j -1 n f (ˆ π ) } = f ( π ) -E { D n,k f (ˆ π ) } . This representation motivates the specific form of D n,k .

Remark 3. In general, the curse of dimensionality may arise and depends on the specific distribution of X and the likelihood function ℓ . There is no universal relationship between n and the dimension d . However, to build intuition, we give an example that illustrates how n and d may relate. Suppose that Y = ( Y (1) , . . . , Y ( d ) ) and X = ( X (1) , . . . , X ( d ) ) have i.i.d. components, and L 1 ≤ P ( Y ( i ) | X ( i ) ) ≤ L 2 for 1 ≤ i ≤ d . Then we have ℓ ( X ) := P ( Y | X ) ∈ [ L d 1 , L d 2 ] . Note that O L 1 ,L 2 ,k ( n -k ) in (5) can be bounded by C ( k )( L d 2 /L d 1 ) 2 k n -k for some constant C ( k ) depending only on k. To ensure that our debiasing method improves over the baseline method without debiasing in the case of growing dimensions, it suffices to let n and d satisfy that ( L d 2 /L d 1 ) 2 k n -k ≪ n -1 when k ≥ 2 , which is equivalent to kd ≪ log( n ) .

Sketch proof for Theorem 1 . First let µ = ∫ X ℓ ( x ) π ( dx ) and µ A = ∫ A ℓ ( x ) π ( dx ) and introduce a new operator

then we have B n D n,k = C n,k . By the definition of B n , it suffices to show that

The first step is to express B j n f ( π ) with the recursive resampled versions of the training data. Specifically, let ˆ π (0) = π, ˆ π (1) = ˆ π and set ( X (0) 1 , . . . , X (0) n ) ≡ ( X 1 , . . . , X n ) . For j = 1 , . . . , k ,

we define ˆ π ( j ) as the empirical measure of n i.i.d. samples ( X ( j -1) 1 , . . . , X ( j -1) n ) drawn from the measure ˆ π ( j -1) . Additionally, let

Then we have

The second step is to rewrite (7) with Taylor expansion of µ ( j -1) A / ( e ( j -1) n + µ ) with respect to e ( j -1) n up to order 2 k -1 . L 1 ≤ l ( X ( j -1) i ) ≤ L 2 and Hoeffding's inequality implies that the expectation of the residual term E { ( e ( j -1) n ) 2 k /ξ 2 k +1 } for some ξ between e ( j -1) n + µ and µ is O L 1 ,L 2 ,k ( n -k ) . Now we instead to show that

since (7) is equal to µ A /µ + ∑ 2 k -1 r =1 ( -1) r µ -r -1 B k,r + O L 1 ,L 2 ,k ( n -k ) . Define a new operator B : h ↦→ E [ h (ˆ π )] for any h : ∆ X → R and let h s ( π ) = { ∫ A ℓ ( x ) π ( dx ) }{ ∫ ℓ ( x ) π ( dx ) } s . Then

The last step is to prove

since (8) is equivalent to ∑ k j =1 ( k j ) ( -1) j -1 B j h s ( π ) = h s ( π )+ O L 1 ,L 2 ,s ( n -k ) . Finally (8) follows from the fact that ( I -B ) k h s ( π ) can be expressed as a finite sum of the terms which have the following form:

where | α a , s ,v | ≤ C k ( s ) n -k for some constnat C k ( s ) (see our Lemma 2).

4.2 Debiased posterior approximation under discrete prior

In this section, we consider the case where X follows a discrete distribution. As mentioned in Remark 2, the result in Theorem 1 is expected to hold in a broader class of samplers f under smoothness, extending beyond just the Bayes-optimal sampler f . The assumption of finite X in this section allows us to simplify some technical aspects in the proof.

Let the support of X be denoted as X = { u 1 , u 2 , . . . , u m } . Assume that |X| = m is finite, and X is distributed according to an unknown prior distribution π ( x ) such that the probability of X taking the value u i is given by π ( X = u i ) = q i for i = 1 , 2 , . . . , m . Here, the probabilities q i are unknown and satisfy the usual constraints that q i ≥ 0 for all i and ∑ m i =1 q i = 1 .

Let q = ( q 1 , · · · , q m ) ⊤ represent the true prior probability vector associated with the probability distribution π ( x ) . Let g be a map from a prior probability vector to a posterior probability vector. Then g ( q ) = ( g 1 ( q ) , · · · , g m ( q )) ⊤ is the probability vector associated with the posterior. Let T = ( T 1 , · · · , T m ) ⊤ where T j = ∑ n i =1 ✶ X i = u j for j = 1 , · · · , m . In such setting, by the definition (4) of operator B n , we can rewrite the operator B n as

where ¯ ∆ m = { ν ∈ N m : ∑ m j =1 ν j = n } and

Additionally, let ∆ m = { q ∈ R m : q j ≥ 0 , ∑ m j =1 q j = 1 } and let ∥ · ∥ C k (∆ m ) denote the C k (∆ m ) -norm which is defined as ∥ f ∥ C k (∆ m ) = ∑ ∥ α ∥ 1 ≤ k ∥ ∂ α f ∥ ∞ for any f ∈ C k (∆ m ) .

Assumption 2. |X| = m is finite, and max s ∈ [ m ] ∥ g s ∥ C 2 k (∆ m ) ≤ G for some constant G.

The following theorem provides a systematic method for constructing an approximation of g s ( q ) with an error of order O ( n -k ) for any desired integer k .

Theorem 2. If |X| = m , let q = ( q 1 , · · · , q m ) ⊤ be the true prior probability vector associated with a discrete probability distribution and T = ( T 1 , · · · , T m ) ⊤ where T j = ∑ n i =1 ✶ X i = u j for j = 1 , · · · , m . Under Assumption 2, the following holds for any s ∈ { 1 , · · · , m } and any k ∈ N + :

Theorem 2 follows directly from the following lemma, which provides the key approximation result.

Lemma 1. For any integers k, m ∈ N + and any function f ∈ C k (∆ m ) , we have

Note that Theorem 2 holds for all mappings g that satisfy Assumption 2. When g represents a mapping corresponding to the Bayes posterior distribution, we know the exact form of g ( q ) . Hence, we can explore sampling schemes for Bayes-optimal mapping g .

We claim that Bayes-optimal mapping g satisfies Assumption 2. In fact, let ℓ s = ℓ ( u s ) := P Y | X ( y ∗ | u s ) . Using Bayes' theorem, the posterior probability of X = u s given y ∗ is given by

In this case, g s ( q ) := ℓ s q s / ∑ m j =1 ℓ j q j for s = 1 , · · · , m . Since |X| = m is finite, we know that there exists a constant c 1 , c 2 > 0 such that c 1 ≤ l j ≤ c 2 for all 1 ≤ j ≤ m , which implies that max s ∈ [ m ] ∥ g s ∥ C 2 k (∆ m ) ≤ G for some constant G based on k .

Moreover, estimating g s ( q ) based on the observations of X n = ( X 1 , · · · , X n ) and y ∗ is sufficient to generate samples from the posterior distribution P X | Y ( u s | y ∗ ) for s = 1 , · · · , m . Since the exact form of g s is known, if we let ˜ P X n ( x = u s | y = y ∗ ) = D n,k ( g s )( T /n ) where T /n denotes the empirical of the training set, we obtain the following theorem.

Theorem 3. Under Assumption 2, for any k ∈ N + , if |X| = m is finite, then there exists an approximate posterior ˜ P X n ( x | y = y ∗ ) satisfies that for any s ∈ { 1 , · · · , m } ,

The proposed sampling scheme in Algorithm 1 generates k samples and a linear combination of whose distributions approximates the posterior. In applications where it is desired to still generate one sample (rather than using a linear combination), we may consider a rejection sampling algorithm based on Theorem 3 to sample from ˜ P X n ( x | y = y ∗ ) . Let T = ( T 1 , · · · , T m ) ⊤ where T j = ∑ n i =1 ✶ X i = u j for j = 1 , · · · , m . Then ( g 1 ( T /n ) , · · · , g m ( T /n ) ) ⊤ is the posterior probability vector associated with the plug-in posterior ̂ P X n ( x | y = y ∗ ) and ( D n,k ( g 1 )( T /n ) , · · · , D n,k ( g m )( T /n ) ) ⊤ is the posterior probability vector associated with the debiased posterior ˜ P X n ( x | y = y ∗ ) . The rejection sampling is described in Algorithm 2.

Algorithm 2 Rejection Sampling for Debiased Posterior ˜ P X n ( x | y = y ∗ )

Input: Plug-in posterior ̂ P X n ( x | y = y ∗ ) , debiased posterior ˜ P X n ( x | y = y ∗ ) , large enough constant M > 0

Output: Sample from the debiased posterior ˜ P X n ( x | y = y ∗ )

1: repeat

2: Sample x ′ ∼ ̂ P X n ( x | y = y ∗ )

3: Sample u ∼ Uniform (0 , M )

4: until u < ˜ P X n ( x ′ | y = y ∗ )

5: return x ′

̂ P X n ( x ′ | y = y ∗ )

In Algorithm 2,

is the upper bound of the ratio of the debiased posterior to the plug-in posterior.

5 Experiments

In this section, we provide numerical experiments to illustrate the debiasing framework for posterior approximation under the binary prior case and the Gaussian mixture prior case.

Binary prior case . Suppose that X = { 0 , 1 } and X ∼ Bern( q ) for some unknown prior q ∈ (0 , 1) . Let α = α ( y ∗ ) := P Y | X ( y ∗ | 1) /P Y | X ( y ∗ | 0) be the likelihood ratio. Then the posterior distribution is give by X | Y ∼ Bern ( αq/ ( αq +1 -q ) ) . We estimate g ( q ) := αq/ ( αq +1 -q ) based on the observations of X n and y ∗ .

Proposition 1 provides a debiased approximation as a special case of Theorem 2 when |X| = 2 .

Proposition 1. Let T = ∑ n i =1 X i . For k = 1 , 2 , 3 , 4 , we have

where D n,k = ∑ k -1 j =0 ( k j +1 ) ( -1) j B j n and B n ( g )( x ) = ∑ n k =0 g ( k n ) ( n k ) x k (1 -x ) n -k is the Bernstein polynomial approximation of g .

In the proof of Theorem 2, we notice that for any k ∈ N + , E X n { D n,k g ( T/n ) } = C n,k g ( q ) , which allows Proposition 1 to be verified in closed form. To validate this result numerically, we consider two parameter settings: in the first experiment we set q = 0 . 4 , y ∗ = 2 , and Y | X ∼ N ( X, 1) , while in the second we set q = 3 / 11 , y ∗ = 1 , and Y | X ∼ N ( X, 1 / 4) .

For both settings, we examine the convergence rate of the debiased estimators D n,k g ( T/n ) for k = 1 , 2 , 3 , 4 . The results are shown in log-log plots in Figure 1, where the vertical axis represents the logarithm of the absolute error and the horizontal axis represents the logarithm of the sample size n . Reference lines with slopes corresponding to n -1 , n -2 , n -3 , and n -4 are included for comparison.

Gaussian mixture prior case . Suppose that X ∼ 1 2 N (0 , 1) + 1 2 N (1 , 1) and Y = X + ξ where ξ ∼ N (0 , 1 / 16) . Additionally, let y ∗ = 0 . 8 and A = { x : x ≥ 0 . 5 } . In this case, we validate the theoretical convergence rate

Since E X n { D n,k f (ˆ π )( A ) } does not have a closed-form expression, we approximate it using Monte Carlo simulation. To ensure that the Monte Carlo error is negligible compared to the bias O ( n -k ) , we select the number of Monte Carlo samples N such that N ≫ n 2 k -1 . In practice, we run simulations for k = 1 and k = 2 and set N = n 3 for k = 1 and N = n 4 for k = 2 .

The results are shown in Figure 2. The figure presents log-log plots where the vertical axis represents the logarithm of the absolute error or of the variance and the horizontal axis represents the logarithm of the sample size n . For both k = 1 and k = 2 , the observed convergence rates align closely with the theoretical predictions.

10 1

10 2

10 3

n (log scale)

10

11

10

9

10

7

10

5

10

3

10

1

Bias (log scale)

X n

{

Dn

, 1

g

(

T

/

n

)}

g

(

q

)

X n

{

Dn

, 2

g

(

T

/

n

)}

g

(

q

)

X n

{

Dn

, 3

g

(

T

/

n

)}

g

(

q

)

X n

{

Dn

, 4

g

(

T

/

n

)}

g

(

q

)

(

n

1 ) Reference

(

n

2 ) Reference

(

n

3 ) Reference

(

n

4 ) Reference

(a)

q

= 0

.

4

, y

∗

= 2

, Y

|

X

∼ N

(

X,

1)

10 1

10 2

10 3

n (log scale)

10

11

10

9

10

7

10

5

10

3

10

1

Bias (log scale)

X n

{

Dn

, 1

g

(

T

/

n

)}

g

(

q

)

X n

{

Dn

, 2

g

(

T

/

n

)}

g

(

q

)

X n

{

Dn

, 3

g

(

T

/

n

)}

g

(

q

)

X n

{

Dn

, 4

g

(

T

/

n

)}

g

(

q

)

(

n

1 ) Reference

(

n

2 ) Reference

(

n

3 ) Reference

(

n

4 ) Reference

(b)

q

= 3

/

11

, y

∗

= 1

, Y

|

X

∼ N

(

X,

1

/

4)

10 1

10 2

n (log scale)

10

4

10

3

10

2

10

1

Bias (log scale)

X n

{

Dn

, 1

f

(

)(

A

)}

f

(

)(

A

)

X n

{

Dn

, 2

f

(

)(

A

)}

f

(

)(

A

)

(

n

1 ) Reference

(

n

2 ) Reference

(a) Bias convergence rate

10 1

10 2

n (log scale)

10

2

10

1

Var

X n

{

Dn

, 1

f

(

)(

A

)}

Var

X n

{

Dn

, 2

f

(

)(

A

)}

(

n

1 ) Reference

(b) Variance convergence rate

6 Conclusion

We introduced a general framework for constructing a debiased posterior approximation through observed samples D and the known likelihood P Y | X when the prior distribution is unknown. Here, a naive strategy that directly plugs the empirical distribution into the Bayes formula or a generative model has a bias, because the likelihood is nonconstant, inducing a distribution shift, and the map from the prior to posterior is nonlinear. It can be shown that the plug-in approach generates ˆ X with bias ∥ E D ( P ˆ X | Y = y ∗ ,D ) -P X | Y = y ∗ ∥ TV = O ( n -1 ) and variance Var D ( P ˆ X | Y = y ∗ ,D ) = O ( n -1 ) . In contrast, our proposed debiasing framework achieves arbitrarily high-order bias rate of O ( n -k ) for any integer k , while maintaining the order of magnitude of the variance. Our framework is black-box in the sense that we only need to resample the training data and feed it into a given black-box conditional generative model. In particular, we provide a rigorous proof for the Bayes-optimal sampler f under the continuous prior setting and for a broad class of samplers f with a continuous 2 k -th derivative, as specified in Assumption 2, under the discrete prior setting. We expect that the proposed debiasing framework could work for general f and will support future developments in bias-corrected posterior estimation and conditional sampling.

Acknowledgments

This research was supported in part by NSF Grant DMS-2515510.

References

Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations , 2023.

Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Representations , 2024.

T. Tony Cai and Mark G. Low. Testing composite hypotheses, hermite polynomials and optimal estimation of a nonsmooth functional. The Annals of Statistics , 39(2):1012-1041, 2011.

Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In USENIX Security Symposium , pages 5253-5270, 2023.

Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations , 2023.

Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences , 117(48):30055-30062, 2020.

Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to-image generation. In Advances in Neural Information Processing Systems , 2023.

Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 9902-9915, 2022.

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems , 2022.

Dong-Hyun Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML , 2013.

Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing , 479:47-59, 2022.

Qiang Liu and Jason Lee. Black-box Importance Sampling. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics , volume 54, pages 952-961, 2017.

Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the 39th International Conference on Machine Learning , 2022.

Sebastian Nowozin. Debiasing evidence approximations: On importance-weighted autoencoders and jackknife variational inference. In International Conference on Learning Representations , 2018.

Chris J. Oates, Mark Girolami, and Nicolas Chopin. Control Functionals for Monte Carlo Integration. Journal of the Royal Statistical Society Series B: Statistical Methodology , 79(3): 695-718, 05 2016.

Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning , pages 26517-26582, 2023.

Maurice Henry Quenouille. Notes on bias in estimation. Biometrika , 43(3-4):353-360, 1956.

Louis Sharrock, Jack Simons, Song Liu, and Mark Beaumont. Sequential neural score estimation: Likelihood-free inference with conditional score based diffusion models. In Proceedings of the 41st International Conference on Machine Learning , volume 235, pages 44565-44602, 2024.

Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Kaizheng Wang. Pseudo-labeling for kernel ridge regression under covariate shift. arXiv preprint arXiv:2302.10160 , 2024.

Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, and Yao Hu. Instantid: Zero-shot identity-preserving generation in seconds. arXiv preprint arXiv:2401.07519 , 2024.

Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical bayes smoothing. In The Thirty Seventh Annual Conference on Learning Theory , pages 49584991, 2024.

Yihong Wu and Pengkun Yang. Polynomial methods in statistical inference: Theory and practice. Foundations and Trends® in Communications and Information Theory , 17(4):402-586, 2020.

Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text-to-image diffusion: Recaptioning, planning, and generating with multimodal llms. In Forty-first International Conference on Machine Learning , 2024.

TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K. Ryu. Diffusion probabilistic models generalize when they fail to memorize. In ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling , 2023.

Hui Yuan, Kaixuan Huang, Chengzhuo Ni, Minshuo Chen, and Mengdi Wang. Reward-directed conditional diffusion: Provable distribution estimation and reward improvement. In Advances in Neural Information Processing Systems , volume 36, pages 60599-60635, 2023.

Andrew Zammit-Mangion, Matthew Sainsbury-Dale, and Raphaël Huser. Neural methods for amortized inference. Annual Review of Statistics and Its Application , 12, 2024.

Kaihong Zhang, Heqi Yin, Feng Liang, and Jingbo Liu. Minimax optimality of score-based diffusion models: Beyond the density lower bound assumptions. In International Conference on Machine Learning , 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The model setting and the assumptions of our claim are clearly stated in abstract and introduction. Our main contributions are stated in introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a black-box debiaisng framework since we only provide the rigorous proof for the Bayes-optimal sampler under continuous prior setting and a broad class of samplers with a continuous 2 k -th derivative under discrete prior setting. We expect the framework could work for general samplers f .

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are clearly stated or referenced in the statement of our theorems and lemmas.The proofs appear in the main paper and the supplemental material.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe our experiments in the Experiment Section in detail.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We used the simulation data in the experiment.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the setting of our simulation in the Experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our experiments do not include any error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: Our experiment is a simple simulation for binary case.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our main contribution is constructing a debiased approximation of the posterior distribution which does not have immediate societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Proof of Theorem 1

Proof of Theorem 1. We begin by introducing notations that facilitates the analysis. Define

Let ˆ π (0) = π, ˆ π (1) = ˆ π and set ( X (0) 1 , . . . , X (0) n ) ≡ ( X 1 , . . . , X n ) . For j = 1 , . . . , k , we define ˆ π ( j ) as the empirical measure of n i.i.d. samples ( X ( j -1) 1 , . . . , X ( j -1) n ) drawn from the measure ˆ π ( j -1) . Furthermore, for each j = 0 , . . . , k , define

Let since B n D n,k = C n,k .

The Radon-Nikodym derivative of f ( π ) with respect to π is

For the empirical measure ˆ π , the corresponding Radon-Nikodym derivative of f (ˆ π ) with respect to ˆ π takes the form

Consequently,

Moreover, by the definition of B n and iterated conditioning, we have E { f (ˆ π ( j ) )( A ) } = E [ E { f (ˆ π ( j ) )( A ) | ˆ π ( j -1) }] = E { B n f (ˆ π ( j -1) )( A ) } = · · · = E { B j -1 n f (ˆ π (1) )( A ) } = B j n f ( π )( A ) .

so that it suffices to show that

By the same logic, for any j = 2 , . . . , k , we have

Thus,

Then (9) holds if

Now we show that (10) holds. By using the Taylor expansion of 1 / ( e ( j -1) n + µ ) , we have

where ξ lies between e ( j -1) n + µ and µ .

Since min { e ( j -1) n + µ, µ } ≥ L 1 , we have 1 /ξ 2 k +1 ≤ L -2 k -1 1 . Additionally, L 1 ≤ l ( X ( j -1) i ) ≤ L 2 and Hoeffding's inequality implies that

for all t > 0 , which is equivalent to

Then

Therefore, we have

which implies that the L.H.S. of (10) can be written as

Thus to prove the bound on the bias, it remains to show that for any given k and 1 ≤ r ≤ 2 k -1 ,

Define a new operator B : h ↦→ E { h (ˆ π ) } for any h : ∆ X → R and let

Since B j h s ( π ) = E { h s (ˆ π ( j ) ) } , we have

We claim that B k,r = O L 1 ,L 2 ,k ( n -k ) holds if for any 0 ≤ s ≤ r ≤ 2 k -1 ,

Indeed, ( I -B ) k h s ( π ) = O L 1 ,L 2 ,s ( n -k ) is equivalent to ∑ k j =1 ( k j ) ( -1) j -1 B j h s ( π ) = h s ( π ) + O L 1 ,L 2 ,s ( n -k ) . Therefore (11) implies

Now, to prove the bound on the bias we only need to show that (11) holds. For any k ∈ N and s ∈ N + , let

and

where C 0 ( s ) , C 1 ( s ) , . . . are constants from Lemma 2. Since h s ( π ) ∈ A 0 s +1 , Lemma 2 implies that ( I -B ) k h s ( π ) ∈ A k s +1 . Therefore, ( I -B ) k h s ( π ) = O L 1 ,L 2 ,s ( n -k ) , finishing the proof for the bias bound.

Finally, to prove the bound on the variance, consider the function F ( x, y ) = x/y . By construction,

Applying the Taylor expansion of F ( x, y ) yields

for some ξ x lying between µ A and µ ( j -1) A , and ξ y lying between µ and e ( j -1) n + µ . Since L 1 ≤ µ A , µ ( j -1) A , µ, e ( j -1) n + µ ≤ L 2 implies that | 1 /ξ 2 y | and | ξ x /ξ 3 y | are bounded by some constant depending on L 1 and L 2 .

Moreover, since

and

Combining these bounds with the Taylor expansion, we conclude that for any j ≥ 1 ,

By the same logic, we also have B n { f ( π )( A ) } 2 = { f ( π )( A ) } 2 + O L 1 ,L 2 ( n -1 ) . Therefore,

and

Lemma 2. There exist constants C 0 ( s ) , C 1 ( s ) , C 2 ( s ) , . . . , such that the following holds. For any k ∈ N and s, n ∈ N + , let

and A k s :=    ∑ ( a , s ,v ) ∈ J s α a , s ,v {∫ A ℓ v ( x ) π ( dx ) } ∏ i {∫ ℓ a i ( x ) π ( dx ) } s i : | α a , s ,v | ≤ C k ( s ) n -k    . If h ( π ) ∈ A 0 s , then for any k ∈ N , we have

where B is an operator defined as Bh ( π ) = E { h (ˆ π ) } where ˆ π is the empirical distribution of X 1 , X 2 , . . . , X n i . i . d . ∼ π .

Proof of Lemma 2. We begin by proving that ( I -B ) h ( π ) ∈ A 1 s . Since h ( π ) ∈ A 0 s , let

Note that | J s | does not depend on n and | α a , s ,v | ≤ C 0 ( s ) . It suffices to verify that each individual term in the sum satisfies

Without loss of generality, let a = ( a 1 , . . . , a p ) and s = ( s 1 , . . . , s p ) , s ′ = ∑ p i =1 s i . Then we have ∑ p i a i s i + v = s and

For the term ∏ p i =1 { ∑ n j =1 ℓ a i ( X j ) } s i , let m ( i ) j denote the times X j appears with powers a i , then we have ∑ n j =1 m ( i ) j = s i for 1 ≤ i ≤ p . Define

Therefore,

where

Note that c s , m does not depend on n . Now we expand R . H . S . of (13) based on the number of distinct variables X j appear, i.e., ∑ n j =1 ∑ p i =1 a i m ( i ) j > 0 , which is equal to ∑ n j =1 ✶ ∑ p i =1 m ( i ) j > 0 . Define then we have 1 ≤ |J | ≤ s ′ .

where c s , m ∗ = ∏ p i =1 s i ! .

The three terms in (14) are interpreted as follows: we can expand { ∑ n t =1 ℓ v ( X t ) δ X t ( A ) } ∏ p i =1 { ∑ n j =1 ℓ a i ( X j ) } s i as the sum of many product terms of the form ℓ v ( X t ) ∏ p i =1 ∏ s i l =1 ℓ a i ( X j i,l ) . The first term in (14) corresponds to the partial sum of terms in which all of X t , ( X j i,l ) i,l are distinct. The second term in (14) corresponds to the partial sum of terms in which X t is identical to one of ( X j i,l ) i,l while the latter are distinct. The third term corresponds to the partial sum of terms in which at least two of ( X j i,l ) i,l are identical. The last two term in (14) are at least O ( n -1 ) factor smaller than the first (due to fewer terms involved in the sum because of the constraint of having identical terms), while the first term will cancel with I · h ( π ) when applying I -B to h .

Let P ( b 1 , . . . , b m ) denote the set of all distinct permutations of the vector consisting of m nonzero elements b 1 , . . . , b m and n -m zeros. Note that even the values of b i may be the same, we still treat the b i s are distinguishable. Then since 0 s are identical, we have |P ( b 1 , . . . , b m ) | = n ( n -1) · · · ( n -m +1) = O ( n m ) . Additionally, for any a and m , we define

Now we can write (14) as

̸

̸

̸

̸

Therefore,

̸

The last inlcusion follows from that fact that { n s ′ +1 -n ( n -1) · · · ( n -s ′ ) } /n s ′ +1 = O ( n -1 ) and the number of solutions to ∑ m k =1 b k = s -v does not depend on n but depends on s .

Now we suppose (12) holds for k . Then we can set

where | α ′ a , s ,v | ≤ C k ( s ) n -k . Then for k +1 , we have

Since for all a , s , v such that ( a , s , v ) ∈ J s , {∫ A ℓ v ( x ) π ( dx ) }∏ i {∫ ℓ a i ( x ) π ( dx ) } s i ∈ A 0 s , we have ( I -B ) {∫ A ℓ v ( x ) π ( dx ) }∏ i {∫ ℓ a i ( x ) π ( dx ) } s i ∈ A 1 s , namely,

where | α b , t ,u ( a , s , v ) | ≤ C 0 ( s ) n -1 . Therefore,

B Proof of Theorem 2

In order to prove Theorem 2, we first make some preliminary observations.

Let function f defined on the simplex ∆ m = { q ∈ R m : q j ≥ 0 , ∑ m j =1 q j = 1 } . Define the generalized Bernstein basis polynomials of degree n as

Proof of Lemma 3. It sufficies to show that ∣ ∣ ∑ ν ∈ ¯ ∆ m ( ν -n q ) α b n, ν ( q ) ∣ ∣ ≲ n ∥ α ∥ 1 / 2 . Since q ∈ ∆ m , we treat T n, α ≡ ∑ ν ∈ ¯ ∆ m ( ν -n q ) α b n, ν ( q ) as a function of the variables q 1 , · · · , q m -1 . For any β ∈ N m -1 such that ∥ β ∥ 1 = 1 , we let γ = γ ( β ) ≡ ( β ⊤ , 0) ⊤ . Additionally, let θ = (0 , · · · , 0 , 1) ⊤ ∈ N m . Since

and we have

i.e.,

By summing the above equation over β ∈ N m -1 such that ∥ β ∥ 1 = 1 , we have

where the last equality follows from the fact that ∑ ∥ β ∥ 1 =1 T n, α + γ + T n, α + θ = 0 . Therefore, we have the following recurrence formula:

Using the recurrence fomrula (15), (16) and the fact that T n, (1 , 0 , ··· , 0) ⊤ = 0 , T n, (2 , 0 , ··· , 0) ⊤ = nq 1 (1 -q 1 ) , T n, (1 , 1 , 0 , ··· , 0) ⊤ = -nq 1 q 2 , T n, α has the following form by using induction:

where c j, η is independent of n . Then we can conclude that | T n, α | ≲ n ⌊∥ α ∥ 1 / 2 ⌋ ≲ n ∥ α ∥ 1 / 2 .

Proof of Lemma 1. We prove the theorem by induction on k .

For k = 1 , by Taylor's expansion, there exists ξ ∈ ∆ m such that

Then we have

where the second inequality follows from Cauchy-Schwarz inequality, and the third inequality follows from Lemma 3.

Suppose the theorem holds up through k . Now we prove the theorem for k +1 . For k +1 , by Taylor's expansion, there exists ξ ∈ ∆ m such that

Then we have

Therefore,

First, we consider the first term of the right-hand side of (18). We know that ( α !) -1 ∂ α f ( q ) ∑ ν ∈ ¯ ∆ m ( ν /n -q ) α b n, ν ( q ) ∈ C k +1 -∥ α ∥ 1 (∆ m ) | since f ∈ C k +1 (∆ m ) . By the induction hypothesis, we have

Let

For any | β | ≤ k +1 -∥ α ∥ 1 , we have

where the last inequality follows from the fact that ∥ ∥ ∂ β -γ {∑ ν ∈ ¯ ∆ m ( ν /n -q ) α b n, ν ( q ) }∥ ∥ ∞ ≲ n -∥ α ∥ 1 / 2 which can be derived by using the form of T n, α in (17).

Therefore, we have

Then we consider the second term of the right-hand side of (18).

Finally, we have

The last inequality holds when ∥ B n -I ∥ ∞ is bounded by a constant independent of n . In fact, ∥ ( B n -I ) f ∥ ∞ = sup q ∈ ∆ m | ( B n -I ) f ( q ) | = sup q ∈ ∆ m ∣ ∣ ∑ ν ∈ ¯ ∆ m { f ( ν /n ) -f ( q ) } b n, ν ( q ) ∣ ∣ ≤ 2 ∥ f ∥ ∞ and ∥ B n -I ∥ ∞ = sup f ∈ C k (∆ m ) , ∥ f ∥ ∞ ≤ 1 ∥ ( B n -I ) f ∥ ∞ ≤ sup f ∈ C k (∆ m ) , ∥ f ∥ ∞ ≤ 1 2 ∥ f ∥ ∞ ≤ 2 .

Proof of Theorem 2. The first claim follows from Lemma 1 and the fact that max s ∈ [ m ] ∥ g s ∥ C 2 k (∆ m ) ≤ G and E X n { D n,k ( g s )( T /n ) } = C n,k ( g s )( q ) . Additionally, we have

and

Therefore,

C Proof of Theorem 3

Proof of Theorem 3. By Theorem 2, it suffices to let ˜ P X n ( x = u s | y = y ∗ ) = D n,k ( g s )( T /n ) . Moreover, we have ∑ m s =1 D n,k ( g s )( T /n ) = 1 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the task of sampling from the posterior distribution $P _ {X|Y=y^*}$ given i.i.d. samples $X _ {1:n}$ from an unknown prior distribution $\\pi$ and a known likelihood model $P _ {Y|X}$. The authors propose a novel approach to reduce the bias of the plug-in approximation (i.e., replacing $\\pi$ with the empirical distribution of the samples) so that the TV distance of the averaged posterior distribution (over the randomness of $X _ {1:n}$) to the true posterior distribution can decay at $O(n^{-k})$ for any integer $k$, while the average variance of the posterior distribution remains $O(n^{-1})$. The authors prove two cases of the continuous and discrete prior distributions, respectively, and validate their theoretical findings through numerical experiments on Bernoulli prior distribution.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThis paper is theoretically solid and well-written. I have gone through the proofs and have not found any significant issues. The notations are mostly clear, though I had some difficulty understanding the definition of $B _ n$ and $B _ n^j$ at first read (but it was clarified in the proof and the pseudocodes). The mathematical derivations are rigorous, and the assumptions are clearly stated. The main results are also clearly presented, with an easy-to-follow sketch of proof in the main text providing good intuition. \n\n**Weaknesses**\n\nWritings can be improved in some places. For example, in typical machine learning conference papers, it is not very common to include heavy mathematical details in the abstract and introduction sections. For instance, content between lines 49 and 71 can be briefly summarized in the introduction but elaborated in the main body. The contribution part can also be summarized and itemized for better clarity. Also, the authors can convert some inline equations to display style for better readability (e.g., lines 385 to 389). I suggest the authors to refer to some well-written papers in top ML conferences for better writing styles.\n\nWhile I understand that this is a statistical theory paper, as the authors propose theoretically-grounded improved methodologies, the empirical results can be further strengthened, such as adding experiments on continuous prior distributions and trying estimators with bias $O(n^{-k})$ for some larger $k$. The rejection sampling proposed in Alg. 2 can also be tried. As the authors mentioned, the proposed framework can be applied to generative models like diffusion models and VAEs for conditional sampling, it would significantly enhance the impact of this work if the authors can demonstrate the effectiveness of their methods on some real-world datasets (i.e., with more complex prior distributions).\n\nQUESTIONS:\nSee weakness part above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLet $(X,Y)$ be a joint distribution, with the likelihood $P_{Y|X}$ assumed known. The paper tackles the task of estimating the posterior $P_{X\\mid Y = y^*}$ from i.i.d. samples $X_1, \\dots, X_n$ drawn from the joint. It concentrates on the **bias** of the estimator, measured as the total-variation distance between the *average* of the estimators and the true posterior. The key technical idea is to apply an operator—borrowed from polynomial-approximation/Richardson-extrapolation ideas—to build an estimator whose bias shrinks as $O(n^{-k})$ while the variance stays at the usual $O(n^{-1})$, for any fixed constant $k$.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. As far as I can tell, the operator-based approach is elegant and may be applicable in a broader context. Although similar ideas might exist (or even be well-known) in approximation theory, their use in the current setting appears novel.  \n2. The results are non-trivial, and the technical proofs are carefully worked out. I did not spot any significant mistakes.\n\n**Weaknesses**\n\n1. My main reservation of the paper is its objective: eliminating the **bias** of the estimator is rather weak. For example, for a single sample $X$ from a distribution $p$ the empirical distribution $\\delta_X$ is completely unbiased. A more relevant metric is the **expected risk** that compares the actual estimate to the underlying truth before taking the outer expectation. Given the variance bound $O(n^{-1})$, the expected risk of the proposed estimator shows no improvement over the naïve plug-in estimator.  Reducing bias relative to the plug-in estimator is not as surprising as the authors suggest.\n2. The guarantees are essentially asymptotic—the constant hidden in the $O(\\cdot)$ notation depends on $k$. A quick check of the proofs suggests it can be as large as $2^k$, which further limits practical usefulness. Can the authors state the constant explicitly, or at least its order?  \n3. The assumption that the likelihood $\\ell(x)$ is bounded away from zero feels unnatural and needs justification.  \n4. In my opinion, this is a classical statistics paper packaged as a machine-learning contribution. I can hardly imagine how the claimed application scenarios (e.g., generative models) match the actual results. A concrete, convincing example would help.\n\n**Overall**\n\nDespite these criticisms, the paper solves a non-trivial problem in a niche area, and the work is solid. I therefore recommend borderline accept.\n\nQUESTIONS:\nSee weakness section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the problem of taking a sample of unconditional observations $X$ and a known data likelihood distribution $P_{Y | X} $ and estimating the posterior distribution $P(X | Y = y^\\ast)$. The approach in this work does not estimate a parametric function for the distribution of $X$ but instead estimates the posterior distribution using a weighted sum of empirical dirac-delta distributions obtained by an iterative data resampling process. The first main result is to present the form of the weighted sum of iterative resampling procedures and to prove that this approximates the true posterior distribution with an approximation error of $\\mathcal{O}(n ^{-k})$ where $n$ is the sample size and $k$ is the number of resampling iterations. Next, a similar theorem with simpler assumptions is proved in the case where $X$ takes on a finite number of discrete values. Two algorithms are presented: one for approximating the posterior distribution using a weighted sum of empirical distribution for $k$ rounds of iterative resampling, and one for drawing a single sample from the approximate posterior in the case where $X$ is discrete. An experimental section validates the theoretical claims in the case where $X$ is a Bernoulli distribution and $Y | X$ follows a Gaussian distribution where the mean depends on $X$ using $k = 2$ resamplings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* The goal of estimating posterior distributions using only unconditional samples and a known likelihood is an important and relevant problem. Providing a method for obtaining better approximations that the naive (and biased) plug-in empirical estimate that only requires straightforward iterative resampling and weighting plug-in estimates has the potential to be a useful and lightweight solution.\n* The level of technical theoretical details is high, and the paper proves several non-trivial results. The theoretical results appear valid from what I reviewed, but the details are very dense and time consuming to thoroughly validate.\n* The experimental results in the Bernoulli/Gaussian setting match very well with the theoretical predictions for $k=2$ resamplings.\n\n**Weaknesses**\n* The experimental section is extremely limited and only examines a single very simple 1D situation. Furthermore, only $k=2$ resamplings are studied, while the theoretical results appear to demonstrate significant benefit for increasingly large $k$. It would be helpful to see experiments on more complex scenarios and for higher $k$.\n* The method relies heavily on the empirical Dirac distribution. While this alleviates the difficulty of learning a parametric model, I would guess that it heavily limits the scalability of the proposed method. In particular, the method seems very susceptible to the curse of dimensionality and prohibitively large number of samples $n$ might be needed in high-dimensional cases. The paper mentions diffusion and other generative models a few times, but I am not sure whether the proposed method is really viable in those kinds of situations.\n\nQUESTIONS:\n* Can more experiments be added for more complex scenarios? How well does this method scale with dimension and distributional complexity?\n* How does the Gaussian/Bernoulli experiment unfold for $k > 2$? Is there eventually a limit to the benefit of increasing $k$? The theoretical results appear to show that increasing $k$ should provide increasing benefits, but I am skeptical that iterative resampling can really provide increasingly accurate estimates since no \"new\" information is really being generated.\n* Can the authors provide an intuitive explanation for the form of $D_{n, k}$? Even after reading through the proof of Theorem 1 a few times, it is difficult to understand why the form $D_{n, k}$ enables better approximation accuracy than the naive plug-in estimator.\n* Does this method overcome limitations of the empirical Dirac distribution in high dimensional settings?\n\nLIMITATIONS:\nLimitations and societal consequences are not discussed by the authors. Lack of discussion about societal consequences does not impact my view of the work. Lack of discussion about limitations does impact my view, in particular limitations that relate to the scalability of the proposed method in terms of distribution dimension and complexity. The method is only applied in a very simple scenario, which makes me think that these limitations might be significant.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}