SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gergely Flamich
One-shot channel simulation is a fundamental data compression problem concerned with encoding a single sample from a target distribution  $Q$  using a coding distribution  $P$  using as few bits as possible on average. Algorithms that solve this problem find applications in neural data compression and differential privacy and can serve as a more efficient alternative to quantization-based methods. Sadly, existing solutions are too slow or have limited applicability, preventing widespread adoption. In this paper, we conclusively solve one-shot channel simulation for one-dimensional problems where the target-proposal density ratio is unimodal by describing an algorithm with optimal runtime. We achieve this by constructing a rejection sampling procedure equivalent to greedily searching over the points of a Poisson process. Hence, we call our algorithm greedy Poisson rejection sampling (GPRS) and analyze the correctness and time complexity of several of its variants. Finally, we empirically verify our theorems, demonstrating that GPRS significantly outperforms the current state-of-the-art method, A* coding. Our code is available at https://github.com/gergely-flamich/greedy-poisson-rejection-sampling.
It is a common misconception that quantization is essential to lossy data compression; in fact, it is merely a way to discard information deterministically. In this paper, we consider the alternative, that is, to discard information stochastically using one-shot channel simulation. To illustrate the main idea, take lossy image compression as an example. Assume we have a generative model given by a joint distribution  $P_{\mathbf{x},\mathbf{y}}$  over images  $\mathbf{y}$  and latent variables  $\mathbf{x}$ , e.g. we might have trained a variational autoencoder (VAE; Kingma & Welling, 2014) on a dataset of images. To compress a new image  $\mathbf{y}$ , we encode a single sample from its posterior  $\mathbf{x} \sim P_{\mathbf{x}|\mathbf{y}}$  as its stochastic lossy representation. The decoder can obtain a lossy reconstruction of  $\mathbf{y}$  by decoding  $\mathbf{x}$  and drawing a sample  $\hat{\mathbf{y}} \sim P_{\mathbf{y}|\mathbf{x}}$  (though in practice, for a VAE we normally just take the mean predicted by the generative network).
Abstracting away from our example, in this paper we will be entirely focused on channel simulation for a pair of correlated random variables  $\mathbf{x},\mathbf{y}\sim P_{\mathbf{x},\mathbf{y}}$ : given a source symbol  $\mathbf{y}\sim P_{\mathbf{y}}$  we wish to encode a single sample  $\mathbf{x}\sim P_{\mathbf{x}|\mathbf{y}}$ . A simple way to achieve this is to encode  $\mathbf{x}$  with entropy coding using the marginal  $P_{\mathbf{x}}$ , whose average coding cost is approximately the entropy  $\mathbb{H}[\mathbf{x}]$ . Surprisingly, however, we can do much better by using a channel simulation protocol, whose average coding cost is approximately the mutual information  $I[\mathbf{x};\mathbf{y}]$  (Li & El Gamal, 2018). This is remarkable, since not only  $I[\mathbf{x};\mathbf{y}]\leq \mathbb{H}[\mathbf{x}]$ , but in many cases  $I[\mathbf{x};\mathbf{y}]$  might be finite even though  $\mathbb{H}[\mathbf{x}]$  is infinite, such as when  $\mathbf{x}$  is continuous. Sadly, most existing protocols place heavy restrictions on  $P_{\mathbf{x},\mathbf{y}}$  or their runtime scales much worse than  $\mathcal{O}(I[\mathbf{x};\mathbf{y}])$ , limiting their practical applicability (Agustsson & Theis, 2020).
In this paper, we propose a family of channel simulation protocols based on a new rejection sampling algorithm, which we can apply to simulate samples from a target distribution  $Q$  using a proposal distribution  $P$  over an arbitrary probability space. The inspiration for our construction comes from an exciting recent line of work which recasts random variate simulation as a search problem over a set
Figure 1: Illustration of three GPRS procedures for a Gaussian target  $Q = \mathcal{N}(1,0.25^2)$  and Gaussian proposal distribution  $P = \mathcal{N}(0,1)$ , with the time axis truncated to the first 17 units. All three variants find the first arrival of the same  $(1,P)$ -Poisson process  $\Pi$  under the graph of  $\varphi = \sigma \circ r$  indicated by the thick dashed black line in each plot. Here,  $r = dQ / dP$  is the target-proposal density ratio, and  $\sigma$  is given by Equation (5). Left: Algorithm 3 sequentially searching through the points of  $\Pi$ . The green circle  $(\bullet)$  shows the first point of  $\Pi$  that falls under  $\varphi$ , and is accepted. All other points are rejected, as indicated by red crosses  $(\times)$ . In practice, Algorithm 3 does not simulate points of  $\Pi$  that arrive after the accepted arrival. Middle: Parallelized GPRS (Algorithm 4) searching through two independent  $(1/2,P)$ -Poisson processes  $\Pi_1$  and  $\Pi_2$  in parallel. Blue points are arrivals in  $\Pi_1$  and orange points are arrivals in  $\Pi_2$ . Crosses  $(\times)$  indicate rejected, and circles  $(\bullet)$  indicate accepted points by each thread. In the end, the algorithm accepts the earliest arrival across all processes, which in this case is marked by the blue circle  $(\bullet)$ . Right: Branch-and-bound GPRS (Algorithm 5), when  $\varphi$  is unimodal. The shaded red areas are never searched or simulated by the algorithm since, given the first two rejections, we know points in those regions cannot fall under  $\varphi$ .
of randomly placed points, specifically a Poisson process (Maddison, 2016). The most well-known examples of sampling-as-search are the Gumbel-max trick and A* sampling (Maddison et al., 2014). Our algorithm, which we call greedy Poisson rejection sampling (GPRS), differs significantly from all previous approaches in terms of what it is searching for, which we can succinctly summarise as: "GPRS searches for the first arrival of a Poisson process II under the graph of an appropriately defined function  $\varphi$ ". The first and simplest variant of GPRS is equivalent to an exhaustive search over all points of II in time order. Next, we show that the exhaustive search is embarrassingly parallelizable, leading to a parallelized variant of GPRS. Finally, when the underlying probability space has more structure, we develop branch-and-bound variants of GPRS that perform a binary search over the points of II. See Figure 1 for an illustration of these three variants.
While GPRS is an interesting sampling algorithm on its own, we also show that each of its variants induces a new one-shot channel simulation protocol. That is, after we receive  $\mathbf{y} \sim P_{\mathbf{y}}$ , we can set  $Q \gets P_{\mathbf{x}|\mathbf{y}}$  and  $P \gets P_{\mathbf{x}}$  and use GPRS to encode a sample  $\mathbf{x} \sim P_{\mathbf{x}|\mathbf{y}}$  at an average bitrate of a little more than the mutual information  $I[\mathbf{x};\mathbf{y}]$ . In particular, on one-dimensional problems where the density ratio  $dP_{\mathbf{x}|\mathbf{y}} / dP_{\mathbf{x}}$  is unimodal for all  $\mathbf{y}$  (which is often the case in practice), branch-and-bound GPRS leads to a protocol with an average runtime of  $\mathcal{O}(I[\mathbf{x};\mathbf{y}])$ , which is optimal. This is a considerable improvement over A* coding (Flamich et al., 2022), the current state-of-the-art method.
- We construct a new rejection sampling algorithm called greedy Poisson rejection sampling, which we can construe as a greedy search over the points of a Poisson process (Algorithm 3). We propose a parallelized (Algorithm 4) and a branch-and-bound variant (Algorithms 5 and 6) of GPRS. We analyze the correctness and runtime of these algorithms.
- We show that each variant of GPRS induces a one-shot channel simulation protocol for correlated random variables  $\mathbf{x},\mathbf{y}\sim P_{\mathbf{x},\mathbf{y}}$  , achieving the optimal average codelength of  $I[\mathbf{x};\mathbf{y}] + \log_2(I[\mathbf{x};\mathbf{y}] + 1) + \mathcal{O}(1)$  bits.
- We prove that when  $\mathbf{x}$  is a  $\mathbb{R}$ -valued random variable and the density ratio  $dP_{\mathbf{x}|\mathbf{y}} / dP_{\mathbf{x}}$  is always unimodal, the channel simulation protocol based on the binary search variant of GPRS achieves  $\mathcal{O}(I[\mathbf{x};\mathbf{y}])$  runtime, which is optimal.
- We conduct toy experiments on one-dimensional problems and show that GPRS compares favourably against  $\mathrm{A}^*$  coding, the current state-of-the-art channel simulation protocol.
The sampling algorithms we construct in this paper are search procedures on randomly placed points in space whose distribution is given by a Poisson process  $\Pi$ . Thus, in this section, we first review the necessary theoretical background for Poisson processes and how we can simulate them on a computer. Then, we formulate standard rejection sampling as a search procedure over the points of a Poisson process to serve as a prototype for our algorithm in the next section. Up to this point, our exposition loosely follows Sections 2, 3 and 5.1 of the excellent work of Maddison (2016), peppered with a few additional results that will be useful for analyzing our algorithm later. Finally, we describe the channel simulation problem, using rejection sampling as a rudimentary solution and describe its shortcomings. This motivates the development of greedy Poisson rejection sampling in Section 3.
2.1 Poisson Processes
A Poisson process  $\Pi$  is a countable collection of random points in some mathematical space  $\Omega$ . In the main text, we will always assume that  $\Pi$  is defined over  $\Omega = \mathbb{R}^{+} \times \mathbb{R}^{d}$  and that all objects involved are measure-theoretically well-behaved for simplicity. For this choice of  $\Omega$ , the positive reals represent time, and  $\mathbb{R}^d$  represents space. However, most results generalize to settings when the spatial domain  $\mathbb{R}^d$  is replaced with some more general space, and we give a general measure-theoretic construction in Appendix A, where the spatial domain is an arbitrary Polish space.
Basic properties of  $\Pi$ : For a set  $A \subseteq \Omega$ , let  $\mathbf{N}(A) \stackrel{\text{def}}{=} |\Pi \cap A|$  denote the number of points of  $\Pi$  falling in the set  $A$ , where  $|\cdot|$  denotes the cardinality of a set. Then,  $\Pi$  is characterized by the following two fundamental properties (Kingman, 1992). First, for two disjoint sets  $A, B \subseteq \Omega$ ,  $A \cap B = \emptyset$ , the number of points of  $\Pi$  that fall in either set are independent random variables:  $\mathbf{N}(A) \perp \mathbf{N}(B)$ .
Second,  $\mathbf{N}(A)$  is Poisson distributed with mean measure  $\mu(A) \stackrel{\text{def}}{=} \mathbb{E}[\mathbf{N}(A)]$ .
Time-ordering the points of  $\Pi$ : Since we assume that  $\Omega = \mathbb{R}^{+} \times \mathbb{R}^{d}$  has a product space structure, we may write the points of  $\Pi$  as a pair of time-space coordinates:  $\Pi = \{(T_{n}, X_{n})\}_{n=1}^{\infty}$ . Furthermore, we can order the points in  $\Pi$  with respect to their time coordinates and index them accordingly, i.e. for  $i < j$  we have  $T_{i} < T_{j}$ . Hence, we refer to  $(T_{n}, X_{n})$  as the  $n$ th arrival of the process.
As a slight abuse of notation, we define  $\mathbf{N}(t) \stackrel{\text{def}}{=} \mathbf{N}([0,t) \times \mathbb{R}^d)$  and  $\mu(t) \stackrel{\text{def}}{=} \mathbb{E}[\mathbf{N}(t)]$ , i.e. these quantities measure the number and average number of points of  $\Pi$  that arrive before time  $t$ , respectively. In this paper, we assume  $\mu(t)$  has derivative  $\mu'(t)$ , and assume for each  $t \geq 0$  there is a conditional probability distribution  $P_{X|T=t}$  with density  $p(x \mid t)$ , such that we can write the mean measure as  $\mu(A) = \int_A p(x \mid t) \mu'(t) dx dt$ .
Simulating  $\Pi$ : A simple method to simulate  $\Pi$  on a computer is to realize it in time order, i.e. at step  $n$ , simulate  $T_{n}$  and then use it to simulate  $X_{n} \sim P_{X|T = T_{n}}$ . We can find the distribution of  $\Pi$ 's first arrival by noting that no point of  $\Pi$  can come before it and hence  $\mathbb{P}[T_1 \geq t] = \mathbb{P}[\mathbf{N}(t) = 0] = \exp(-\mu(t))$ , where the second equality follows since  $\mathbf{N}(t)$  is Poisson distributed with mean  $\mu(t)$ . A particularly important case is when  $\Pi$  is time-homogeneous, i.e.  $\mu(t) = \lambda t$  for some  $\lambda > 0$  in which case  $T_{1} \sim \mathrm{Exp}(\lambda)$  is an exponential random variable with rate  $\lambda$ . In fact, all of  $\Pi$ 's inter-arrival times  $\Delta_{n} = T_{n} - T_{n-1}$  for  $n \geq 1$  share this simple distribution, where we set  $T_{0} = 0$ . To see this, note that
Algorithm 1: Generating a (λ, PXT)-Poisson process.
Input : Time rate λ, Spatial distribution PX|T
T0← 0
for n = 1, 2, ... do
Δn ~ Exp(λ)
Tn← Tn-1 + Δn
Xn ~ PX|T=Tr
yield (Tn, Xn)
end
$$
\mathbb {P} \left[ \Delta_ {n} \geq t \mid T _ {n - 1} \right] = \mathbb {P} \left[ \mathbf {N} \left(\left[ T _ {n - 1}, T _ {n - 1} + t\right) \times \mathbb {R} ^ {d}\right) = 0 \mid T _ {n - 1} \right] = \exp (- \lambda t), \tag {1}
$$
i.e. all  $\Delta_n \mid T_{n-1} \sim \mathrm{Exp}(\lambda)$ . Therefore, we can use the above procedure to simulate time-homogeneous Poisson processes, described in Algorithm 1. We will refer to a time-homogeneous Poisson process with time rate  $\lambda$  and spatial distribution  $P_{X|T}$  as a  $(\lambda, P_{X|T})$ -Poisson process.
Rejection sampling using  $\Pi$ : Rejection sampling is a technique to simulate samples from a target distribution  $Q$  using a proposal distribution  $P$ , assuming we can find an upper bound  $M > 0$  for
Algorithm 2: Standard rejection sampler.	Algorithm 3: Greedy Poisson rejection sampler.
Input : Proposal distribution P,
Density ratio r = dQ/dP,
Upper bound M for r.	Input : Proposal distribution P,
Density ratio r = dQ/dP,
Stretch function σ.
Output:Sample X ~ Q and its index N.
// Generator for a (1,P)-Poisson
process using Algorithm 1.
Π ← SimulatePP(1,P)
for n = 1,2,... do
(Tn,Xn) ← next(Π)
Unn ~ Unif(0,1)
if Un < r(Xn)/M then
| return Xn,n
end
end	Output:Sample X ~ Q and its index N.
// Generator for a (1,P)-Poisson
process using Algorithm 1.
Π ← SimulatePP(1,P)
for n = 1,2,... do
(Tn,Xn) ← next(Π)
if Tn < σ(r(Xn)) then
| return Xn,n
end
end
their density ratio  $r = dQ / dP$  (technically, the Radon-Nikodym derivative). We can formulate this procedure using a Poisson process: we simulate the arrivals  $(T_{n},X_{n})$  of a  $(1,P)$ -Poisson process  $\Pi$ , but we only keep them with probability  $r(X_n) / M$ , otherwise, we delete them. This algorithm is described in Algorithm 2; its correctness is guaranteed by the thinning theorem (Maddison, 2016).
Rejection sampling is suboptimal: Using Poisson processes to formulate rejection sampling highlights a subtle but crucial inefficiency: it does not make use of  $\Pi$ 's temporal structure and only uses the spatial coordinates. GPRS fixes this by using a rejection criterion that does depend on the time variable. As we show, this significantly speeds up sampling for certain classes of distributions.
2.2 Channel Simulation
The main motivation for our work is to develop a one-shot channel simulation protocol using the sampling algorithm we derive in Section 3. Channel simulation is of significant theoretical and practical interest. Recent works used it to compress neural network weights, achieving state-of-the-art performance (Havasi et al., 2018); to perform image compression using variational autoencoders (Flamich et al., 2020) and diffusion models with perfect realism (Theis et al., 2022); and to perform differentially private federated learning by compressing noisy gradients (Shah et al., 2022).
One-shot channel simulation is a communication problem between two parties, Alice and Bob, sharing a joint distribution  $P_{\mathbf{x},\mathbf{y}}$  over two correlated random variables  $\mathbf{x}$  and  $\mathbf{y}$ , where we assume that Alice and Bob can simulate samples from the marginal  $P_{\mathbf{x}}$ . In a single round of communication, Alice receives a sample  $\mathbf{y} \sim P_{\mathbf{y}}$  from the marginal distribution over  $\mathbf{y}$ . Then, she needs to send the minimum number of bits to Bob such that he can simulate a single sample from the conditional distribution  $\mathbf{x} \sim P_{\mathbf{x}|\mathbf{y}}$ . Note that Bob does not want to learn  $P_{\mathbf{x}|\mathbf{y}}$ ; he just wants to simulate a single sample from it. Surprisingly, when Alice and Bob have access to shared randomness, e.g. by sharing the seed of their random number generator before communication, they can solve channel simulation very efficiently. Mathematically, in this paper we will always model this shared randomness by some time-homogeneous Poisson process (or processes)  $\Pi$ , since given a shared random seed, Alice and Bob can always simulate the same process using Algorithm 1. Then, the average coding cost of  $\mathbf{x}$  given  $\Pi$  is its conditional entropy  $\mathbb{H}[\mathbf{x} \mid \Pi]$  and, surprisingly, it is always upper bounded by  $\mathbb{H}[\mathbf{x} \mid \Pi] \leq I[\mathbf{x};\mathbf{y}] + \log_2(I[\mathbf{x};\mathbf{y}] + 1) + O(1)$ , where  $I[\mathbf{x};\mathbf{y}]$  is the mutual information between  $\mathbf{x}$  and  $\mathbf{y}$  (Li & El Gamal, 2018). This is an especially curious result, given that in many cases  $\mathbb{H}[\mathbf{x}]$  is infinite while  $I[\mathbf{x};\mathbf{y}]$  is finite, e.g. when  $\mathbf{x}$  is a continuous variable. In essence, this result means that given the additional structure  $P_{\mathbf{x},\mathbf{y}}$ , channel simulation protocols can "offload" an infinite amount of information into the shared randomness, and only communicate the finitely many "necessary" bits.
An example channel simulation protocol with rejection sampling: Given  $\Pi$  and  $\mathbf{y} \sim P_{\mathbf{y}}$ , Alice sets  $Q \gets P_{\mathbf{x}|\mathbf{y}}$  as the target and  $P \gets P_{\mathbf{x}}$  as the proposal distribution with density ratio  $r = dQ / dP$ , and run the rejection sampler in Algorithm 2 to find the first point of  $\Pi$  that was not deleted. She counts the number of samples  $N$  she had to simulate before acceptance and sends this number to Bob. He can then decode a sample  $\mathbf{x} \sim P_{\mathbf{x}|\mathbf{y}}$  by selecting the spatial coordinate of the  $N$ th arrival of  $\Pi$ .
Unfortunately, this simple protocol is suboptimal. To see this, let  $D_{\infty}[Q \| P] \stackrel{\text{def}}{=} \sup_{\mathbf{x} \in \Omega} \{\log_2 r(\mathbf{x})\}$  denote Rényi  $\infty$ -divergence of  $Q$  from  $P$ , and recall two standard facts: (1) the best possible upper bound Alice can use for rejection sampling is  $M_{opt} = \exp_2(D_{\infty}[Q \| P])$ , where  $\exp_2(x) = 2^x$ , and (2), the number of samples  $N$  drawn until acceptance is a geometric random variable with mean  $M_{opt}$  (Maddison, 2016). We now state the two issues with rejection sampling that GPRS solves.
Problem 1: Codelength. By using the formula for the entropy of a geometric random variable and assuming Alice uses the best possible bound  $M_{opt}$  in the protocol, we find that
$$
\mathbb {H} [ \mathbf {x} \mid \Pi ] = \mathbb {E} _ {\mathbf {y} \sim P _ {\mathbf {y}}} [ \mathbb {H} [ N \mid \mathbf {y} ] ] \geq \mathbb {E} _ {\mathbf {y} \sim P _ {\mathbf {y}}} [ D _ {\infty} [ P _ {\mathbf {x} | \mathbf {y}} \| P _ {\mathbf {x}} ] ] \overset {(a)} {\geq} I [ \mathbf {x}; \mathbf {y} ], \tag {2}
$$
see Appendix I for the derivation. Unfortunately, inequality (a) can be arbitrarily loose, hence the average codelength scales with the expected  $\infty$ -divergence instead of  $I[\mathbf{x};\mathbf{y}]$ , as would be optimal.
Problem 2: Slow runtime. We are interested in classifying the time complexity of our protocol. As we saw, for a target  $Q$  and proposal  $P$ , Algorithm 2 draws  $M_{opt} = \exp_2(D_\infty [Q\| P])$  samples on average. Unfortunately, under the computational hardness assumption  $\mathrm{RP} \neq \mathrm{NP}$ , Agustsson & Theis (2020) showed that without any further assumptions, there is no sampler that scales polynomially in  $D_{\mathrm{KL}}[Q\| P]$ . However, with further assumptions, we can do much better, as we show in Section 3.1.
3 Greedy Poisson Rejection Sampling
We now describe GPRS; its pseudo-code is shown in Algorithm 3. This section assumes that  $Q$  and  $P$  are the target and proposal distributions, respectively, and  $r = dQ / dP$  is their density ratio. Let  $\Pi$  be a  $(1, P)$ -Poisson process. Our proposed rejection criterion is now embarrassingly simple: for an appropriate invertible function  $\sigma : \mathbb{R}^{+} \to \mathbb{R}^{+}$ , accept the first arrival of  $\Pi$  that falls under the graph of the composite function  $\varphi = \sigma \circ r$ , as illustrated in the left plot in Figure 1. We refer to  $\sigma$  as the stretch function for  $r$ , as its purpose is to stretch the density ratio along the time-axis towards  $\infty$ .
Deriving the stretch function (sketch, see Appendix A for details): Let  $\varphi = \sigma \circ r$ , where for now  $\sigma$  is an arbitrary invertible function on  $\mathbb{R}^+$ , let  $U = \{(t,x) \in \Omega \mid t \leq \varphi(x)\}$  be the set of points under the graph of  $\varphi$  and let  $\tilde{\Pi} = \Pi \cap U$ . By the restriction theorem (Kingman, 1992),  $\tilde{\Pi}$  is also a Poisson process with mean measure  $\tilde{\mu}(A) = \mu(A \cap U)$ . Let  $(\tilde{T},\tilde{X})$  be the first arrival of  $\tilde{\Pi}$ , i.e. the first arrival of  $\Pi$  under  $\varphi$  and let  $Q_{\sigma}$  be the marginal distribution of  $\tilde{X}$ . Then, as we show in Appendix A, the density ratio  $dQ_{\sigma} / dP$  is given by
$$
\frac {d Q _ {\sigma}}{d P} (x) = \int_ {0} ^ {\varphi (x)} \mathbb {P} [ \tilde {T} \geq t ] d t. \tag {3}
$$
Now we pick  $\sigma$  such that  $Q_{\sigma} = Q$ , for which we need to ensure that  $dQ_{\sigma} / dP = r$ . Substituting  $t = \varphi(x)$  into Equation (3), and differentiating, we get  $(\sigma^{-1})'(t) = \mathbb{P}[\tilde{T} \geq t]$ . Since  $(\tilde{T}, \tilde{X})$  falls under the graph of  $\varphi$  by definition, we have  $\mathbb{P}[\tilde{T} \geq t] = \mathbb{P}[\tilde{T} \geq t, r(\tilde{X}) \geq \sigma^{-1}(t)]$ . By expanding the definition of the right-hand side, we obtain a time-invariant ODE for  $\sigma^{-1}$ :
$$
\left(\sigma^ {- 1}\right) ^ {\prime} = w _ {Q} \left(\sigma^ {- 1}\right) - \sigma^ {- 1} \cdot w _ {P} \left(\sigma^ {- 1}\right), \quad \text {w i t h} \quad \sigma^ {- 1} (0) = 0, \tag {4}
$$
where  $w_{P}(h)\stackrel {def}{=}\mathbb{P}_{Z\sim P}[r(Z)\geq h]$  and  $w_{Q}$  defined analogously. Finally, solving the ODE, we get
$$
\sigma (h) = \int_ {0} ^ {h} \frac {1}{w _ {Q} (\eta) - \eta \cdot w _ {P} (\eta)} d \eta . \tag {5}
$$
Remember that picking  $\sigma$  according to Equation (5) ensures that GPRS is correct by construction. To complete the picture, in Appendix A we prove that  $(\tilde{T},\tilde{X})$  always exists and Algorithm 3 terminates with probability 1. We now turn our attention to analyzing the runtime of Algorithm 3 and surprisingly, we find that the expected runtime of GPRS matches that of standard rejection sampling; see Appendix B.2 for the proof. Note that in our analyses, we identify GPRS's time complexity with the number of arrivals of  $\Pi$  it simulates. Moreover, we assume that we can evaluate  $\sigma$  in  $\mathcal{O}(1)$  time.
Theorem 3.1 (Expected Runtime). Let  $Q$  and  $P$  be the target and proposal distributions for Algorithm 3, respectively, and  $r = dQ / dP$  their density ratio. Let  $N$  denote the number of samples simulated by the algorithm before it terminates. Then,
$$
\mathbb {E} [ N ] = \exp_ {2} \left(D _ {\infty} [ Q \| P ]\right) \quad a n d \quad \mathbb {V} [ N ] \geq \exp_ {2} \left(D _ {\infty} [ Q \| P ]\right), \tag {6}
$$
where  $\mathbb{V}[\cdot ]$  denotes the variance of a random variable.
Furthermore, using ideas from the work of Liu & Verdu (2018), we can show the following upper bound on the fractional moments of the index; see Appendix B.3 for the proof.