Title: Greedy Poisson Rejection Sampling
Venue: NeurIPS 2023
Accepted: yes
Keywords: channel simulation

relative entropy coding

reverse channel coding

rejection sampling

Poisson process
Abstract: One-shot channel simulation is a fundamental data compression problem concerned with encoding a single sample from a target distribution $Q$ using a coding distribution $P$ using as few bits as possible on average. Algorithms that solve this problem find applications in neural data compression and differential privacy and can serve as a more efficient and natural alternative to quantization-based methods. Unfortunately, existing solutions are too slow or have limited applicability, preventing their widespread adaptation. In this paper, we conclusively solve one-shot channel simulation for one-dimensional problems where the target-proposal density ratio is unimodal by describing an algorithm with optimal runtime. We achieve this by constructing a rejection sampling procedure equivalent to greedily searching over the points of a Poisson process. Hence, we call our algorithm greedy Poisson rejection sampling (GPRS) and analyze the correctness and time complexity of several of its variants. Finally, we empirically verify our theorems, demonstrating that GPRS significantly outperforms the current state-of-the-art method, A* coding.

Greedy Poisson Rejection Sampling

Abstract

One-shot channel simulation is a fundamental data compression problem concerned with encoding a single sample from a target distribution Q using a coding distribution P using as few bits as possible on average. Algorithms that solve this problem find applications in neural data compression and differential privacy and can serve as a more efficient alternative to quantization-based methods. Sadly, existing solutions are too slow or have limited applicability, preventing widespread adoption. In this paper, we conclusively solve one-shot channel simulation for one-dimensional problems where the target-proposal density ratio is unimodal by describing an algorithm with optimal runtime. We achieve this by constructing a rejection sampling procedure equivalent to greedily searching over the points of a Poisson process. Hence, we call our algorithm greedy Poisson rejection sampling (GPRS) and analyze the correctness and time complexity of several of its variants. Finally, we empirically verify our theorems, demonstrating that GPRS significantly outperforms the current state-of-the-art method, A* coding. Our code is available at https:// github.com/gergely-flamich/greedy-poisson-rejection-sampling .

1 Introduction

It is a common misconception that quantization is essential to lossy data compression; in fact, it is merely a way to discard information deterministically. In this paper, we consider the alternative, that is, to discard information stochastically using one-shot channel simulation . To illustrate the main idea, take lossy image compression as an example. Assume we have a generative model given by a joint distribution P x , y over images y and latent variables x , e.g. we might have trained a variational autoencoder (VAE; Kingma & Welling, 2014) on a dataset of images. To compress a new image y , we encode a single sample from its posterior x ∼ P x | y as its stochastic lossy representation. The decoder can obtain a lossy reconstruction of y by decoding x and drawing a sample ˆ y ∼ P y | x (though in practice, for a V AE we normally just take the mean predicted by the generative network).

Abstracting away from our example, in this paper we will be entirely focused on channel simulation for a pair of correlated random variables x , y ∼ P x , y : given a source symbol y ∼ P y we wish to encode a single sample x ∼ P x | y . A simple way to achieve this is to encode x with entropy coding using the marginal P x , whose average coding cost is approximately the entropy H [ x ] . Surprisingly, however, we can do much better by using a channel simulation protocol , whose average coding cost is approximately the mutual information I [ x ; y ] (Li & El Gamal, 2018). This is remarkable, since not only I [ x ; y ] ≤ H [ x ] , but in many cases I [ x ; y ] might be finite even though H [ x ] is infinite, such as when x is continuous. Sadly, most existing protocols place heavy restrictions on P x , y or their runtime scales much worse than O ( I [ x ; y ]) , limiting their practical applicability (Agustsson & Theis, 2020).

In this paper, we propose a family of channel simulation protocols based on a new rejection sampling algorithm, which we can apply to simulate samples from a target distribution Q using a proposal distribution P over an arbitrary probability space. The inspiration for our construction comes from an exciting recent line of work which recasts random variate simulation as a search problem over a set

-

1

0

1

0

5

10

15

X

T

-

1

0

1

X

-

1

0

1

X

of randomly placed points, specifically a Poisson process (Maddison, 2016). The most well-known examples of sampling-as-search are the Gumbel-max trick and A* sampling (Maddison et al., 2014). Our algorithm, which we call greedy Poisson rejection sampling (GPRS), differs significantly from all previous approaches in terms of what it is searching for, which we can succinctly summarise as: 'GPRS searches for the first arrival of a Poisson process Π under the graph of an appropriately defined function ϕ '. The first and simplest variant of GPRS is equivalent to an exhaustive search over all points of Π in time order. Next, we show that the exhaustive search is embarrassingly parallelizable, leading to a parallelized variant of GPRS. Finally, when the underlying probability space has more structure, we develop branch-and-bound variants of GPRS that perform a binary search over the points of Π . See Figure 1 for an illustration of these three variants.

While GPRS is an interesting sampling algorithm on its own, we also show that each of its variants induces a new one-shot channel simulation protocol. That is, after we receive y ∼ P y , we can set Q ← P x | y and P ← P x and use GPRS to encode a sample x ∼ P x | y at an average bitrate of a little more than the mutual information I [ x ; y ] . In particular, on one-dimensional problems where the density ratio dP x | y /dP x is unimodal for all y (which is often the case in practice), branch-andbound GPRS leads to a protocol with an average runtime of O ( I [ x ; y ]) , which is optimal. This is a considerable improvement over A* coding (Flamich et al., 2022), the current state-of-the-art method.

In summary, our contributions are as follows:

We construct a new rejection sampling algorithm called greedy Poisson rejection sampling , which we can construe as a greedy search over the points of a Poisson process (Algorithm 3). We propose a parallelized (Algorithm 4) and a branch-and-bound variant (Algorithms 5 and 6) of GPRS. We analyze the correctness and runtime of these algorithms.

We show that each variant of GPRS induces a one-shot channel simulation protocol for correlated random variables x , y ∼ P x , y , achieving the optimal average codelength of I [ x ; y ] + log 2 ( I [ x ; y ] + 1) + O (1) bits.

We prove that when x is a R -valued random variable and the density ratio dP x | y /dP x is always unimodal, the channel simulation protocol based on the binary search variant of GPRS achieves O ( I [ x ; y ]) runtime, which is optimal.

We conduct toy experiments on one-dimensional problems and show that GPRS compares favourably against A* coding, the current state-of-the-art channel simulation protocol.

2 Background

The sampling algorithms we construct in this paper are search procedures on randomly placed points in space whose distribution is given by a Poisson process Π . Thus, in this section, we first review the necessary theoretical background for Poisson processes and how we can simulate them on a computer. Then, we formulate standard rejection sampling as a search procedure over the points of a Poisson process to serve as a prototype for our algorithm in the next section. Up to this point, our exposition loosely follows Sections 2, 3 and 5.1 of the excellent work of Maddison (2016), peppered with a few additional results that will be useful for analyzing our algorithm later. Finally, we describe the channel simulation problem, using rejection sampling as a rudimentary solution and describe its shortcomings. This motivates the development of greedy Poisson rejection sampling in Section 3.

2.1 Poisson Processes

A Poisson process Π is a countable collection of random points in some mathematical space Ω . In the main text, we will always assume that Π is defined over Ω = R + × R d and that all objects involved are measure-theoretically well-behaved for simplicity. For this choice of Ω , the positive reals represent time , and R d represents space . However, most results generalize to settings when the spatial domain R d is replaced with some more general space, and we give a general measure-theoretic construction in Appendix A, where the spatial domain is an arbitrary Polish space.

Basic properties of Π : For a set A ⊆ Ω , let N ( A ) def = | Π ∩ A | denote the number of points of Π falling in the set A , where |·| denotes the cardinality of a set. Then, Π is characterized by the following two fundamental properties (Kingman, 1992). First, for two disjoint sets A,B ⊆ Ω , A ∩ B = ∅ , the number of points of Π that fall in either set are independent random variables: N ( A ) ⊥ N ( B ) .

Time-ordering the points of Π : Since we assume that Ω = R + × R d has a product space structure, we may write the points of Π as a pair of time-space coordinates: Π = { ( T n , X n ) } ∞ n =1 . Furthermore, we can order the points in Π with respect to their time coordinates and index them accordingly, i.e. for i < j we have T i < T j . Hence, we refer to ( T n , X n ) as the n th arrival of the process.

As a slight abuse of notation, we define N ( t ) def = N ([0 , t ) × R d ) and µ ( t ) def = E [ N ( t )] , i.e. these quantities measure the number and average number of points of Π that arrive before time t , respectively. In this paper, we assume µ ( t ) has derivative µ ′ ( t ) , and assume for each t ≥ 0 there is a conditional probability distribution P X | T = t with density p ( x | t ) , such that we can write the mean measure as µ ( A ) = ∫ A p ( x | t ) µ ′ ( t ) dxdt .

Simulating Π : A simple method to simulate Π on a computer is to realize it in time order, i.e. at step n , simulate T n and then use it to simulate X n ∼ P X | T = T n . We can find the distribution of Π 's first arrival by noting that no point of Π can come before it and hence P [ T 1 ≥ t ] = P [ N ( t ) = 0] = exp( -µ ( t )) , where the second equality follows since N ( t ) is Poisson distributed with mean µ ( t ) . A particularly important case is when Π is time-homogeneous , i.e. µ ( t ) = λt for some λ > 0 , in which case T 1 ∼ Exp( λ ) is an exponential random variable with rate λ . In fact, all of Π 's inter-arrival times ∆ n = T n -T n -1 for n ≥ 1 share this simple distribution, where we set T 0 = 0 . To see this, note that

Algorithm 1: Generating a ( λ, P ) -Poisson process.

i.e. all ∆ n | T n -1 ∼ Exp( λ ) . Therefore, we can use the above procedure to simulate timehomogeneous Poisson processes, described in Algorithm 1. We will refer to a time-homogeneous Poisson process with time rate λ and spatial distribution P X | T as a ( λ, P X | T ) -Poisson process.

Rejection sampling using Π : Rejection sampling is a technique to simulate samples from a target distribution Q using a proposal distribution P , assuming we can find an upper bound M > 0 for

their density ratio r = dQ/dP (technically, the Radon-Nikodym derivative). We can formulate this procedure using a Poisson process: we simulate the arrivals ( T n , X n ) of a (1 , P ) -Poisson process Π , but we only keep them with probability r ( X n ) /M , otherwise, we delete them. This algorithm is described in Algorithm 2; its correctness is guaranteed by the thinning theorem (Maddison, 2016).

Rejection sampling is suboptimal: Using Poisson processes to formulate rejection sampling highlights a subtle but crucial inefficiency: it does not make use of Π 's temporal structure and only uses the spatial coordinates. GPRS fixes this by using a rejection criterion that does depend on the time variable. As we show, this significantly speeds up sampling for certain classes of distributions.

2.2 Channel Simulation

The main motivation for our work is to develop a one-shot channel simulation protocol using the sampling algorithm we derive in Section 3. Channel simulation is of significant theoretical and practical interest. Recent works used it to compress neural network weights, achieving state-of-the-art performance (Havasi et al., 2018); to perform image compression using variational autoencoders (Flamich et al., 2020) and diffusion models with perfect realism (Theis et al., 2022); and to perform differentially private federated learning by compressing noisy gradients (Shah et al., 2022).

One-shot channel simulation is a communication problem between two parties, Alice and Bob, sharing a joint distribution P x , y over two correlated random variables x and y , where we assume that Alice and Bob can simulate samples from the marginal P x . In a single round of communication, Alice receives a sample y ∼ P y from the marginal distribution over y . Then, she needs to send the minimum number of bits to Bob such that he can simulate a single sample from the conditional distribution x ∼ P x | y . Note that Bob does not want to learn P x | y ; he just wants to simulate a single sample from it. Surprisingly, when Alice and Bob have access to shared randomness , e.g. by sharing the seed of their random number generator before communication, they can solve channel simulation very efficiently. Mathematically, in this paper we will always model this shared randomness by some time-homogeneous Poisson process (or processes) Π , since given a shared random seed, Alice and Bob can always simulate the same process using Algorithm 1. Then, the average coding cost of x given Π is its conditional entropy H [ x | Π] and, surprisingly, it is always upper bounded by H [ x | Π] ≤ I [ x ; y ] + log 2 ( I [ x ; y ] + 1) + O (1) , where I [ x ; y ] is the mutual information between x and y (Li & El Gamal, 2018). This is an especially curious result, given that in many cases H [ x ] is infinite while I [ x ; y ] is finite, e.g. when x is a continuous variable. In essence, this result means that given the additional structure P x , y , channel simulation protocols can 'offload' an infinite amount of information into the shared randomness, and only communicate the finitely many 'necessary' bits.

An example channel simulation protocol with rejection sampling: Given Π and y ∼ P y , Alice sets Q ← P x | y as the target and P ← P x as the proposal distribution with density ratio r = dQ/dP , and run the rejection sampler in Algorithm 2 to find the first point of Π that was not deleted. She counts the number of samples N she had to simulate before acceptance and sends this number to Bob. He can then decode a sample x ∼ P x | y by selecting the spatial coordinate of the N th arrival of Π .

Unfortunately, this simple protocol is suboptimal. To see this, let D ∞ [ Q ‖ P ] def = sup x ∈ Ω { log 2 r ( x ) } denote Rényi ∞ -divergence of Q from P , and recall two standard facts: (1) the best possible upper bound Alice can use for rejection sampling is M opt = exp 2 ( D ∞ [ Q ‖ P ]) , where exp 2 ( x ) = 2 x , and (2), the number of samples N drawn until acceptance is a geometric random variable with mean M opt (Maddison, 2016). We now state the two issues with rejection sampling that GPRS solves.

Problem 1: Codelength. By using the formula for the entropy of a geometric random variable and assuming Alice uses the best possible bound M opt in the protocol, we find that

see Appendix I for the derivation. Unfortunately, inequality (a) can be arbitrarily loose , hence the average codelength scales with the expected ∞ -divergence instead of I [ x ; y ] , as would be optimal.

glyph[negationslash]

Problem 2: Slow runtime. We are interested in classifying the time complexity of our protocol. As we saw, for a target Q and proposal P , Algorithm 2 draws M opt = exp 2 ( D ∞ [ Q ‖ P ]) samples on average. Unfortunately, under the computational hardness assumption RP = NP , Agustsson & Theis (2020) showed that without any further assumptions, there is no sampler that scales polynomially in D KL [ Q ‖ P ] . However, with further assumptions, we can do much better, as we show in Section 3.1.

3 Greedy Poisson Rejection Sampling

We now describe GPRS; its pseudo-code is shown in Algorithm 3. This section assumes that Q and P are the target and proposal distributions, respectively, and r = dQ/dP is their density ratio. Let Π be a (1 , P ) -Poisson process. Our proposed rejection criterion is now embarrassingly simple: for an appropriate invertible function σ : R + → R + , accept the first arrival of Π that falls under the graph of the composite function ϕ = σ ◦ r , as illustrated in the left plot in Figure 1. We refer to σ as the stretch function for r , as its purpose is to stretch the density ratio along the time-axis towards ∞ .

Deriving the stretch function (sketch, see Appendix A for details): Let ϕ = σ ◦ r , where for now σ is an arbitrary invertible function on R + , let U = { ( t, x ) ∈ Ω | t ≤ ϕ ( x ) } be the set of points under the graph of ϕ and let ˜ Π = Π ∩ U . By the restriction theorem (Kingman, 1992), ˜ Π is also a Poisson process with mean measure ˜ µ ( A ) = µ ( A ∩ U ) . Let ( ˜ T, ˜ X ) be the first arrival of ˜ Π , i.e. the first arrival of Π under ϕ and let Q σ be the marginal distribution of ˜ X . Then, as we show in Appendix A, the density ratio dQ σ /dP is given by

Now we pick σ such that Q σ = Q , for which we need to ensure that dQ σ /dP = r . Substituting t = ϕ ( x ) into Equation (3), and differentiating, we get ( σ -1 ) ′ ( t ) = P [ ˜ T ≥ t ] . Since ( ˜ T, ˜ X ) falls under the graph of ϕ by definition, we have P [ ˜ T ≥ t ] = P [ ˜ T ≥ t, r ( ˜ X ) ≥ σ -1 ( t )] . By expanding the definition of the right-hand side, we obtain a time-invariant ODE for σ -1 :

where w P ( h ) def = P Z ∼ P [ r ( Z ) ≥ h ] and w Q defined analogously. Finally, solving the ODE, we get

Remember that picking σ according to Equation (5) ensures that GPRS is correct by construction . To complete the picture, in Appendix A we prove that ( ˜ T, ˜ X ) always exists and Algorithm 3 terminates with probability 1 . We now turn our attention to analyzing the runtime of Algorithm 3 and surprisingly, we find that the expected runtime of GPRS matches that of standard rejection sampling; see Appendix B.2 for the proof. Note that in our analyses, we identify GPRS's time complexity with the number of arrivals of Π it simulates. Moreover, we assume that we can evaluate σ in O (1) time.

Theorem 3.1 (Expected Runtime) . Let Q and P be the target and proposal distributions for Algorithm 3, respectively, and r = dQ/dP their density ratio. Let N denote the number of samples simulated by the algorithm before it terminates. Then,

where V [ · ] denotes the variance of a random variable.

Furthermore, using ideas from the work of Liu & Verdu (2018), we can show the following upper bound on the fractional moments of the index; see Appendix B.3 for the proof.

Theorem 3.2 (Fractional Moments of the Index) . Let Q and P be the target and proposal distributions for Algorithm 3, respectively, and r = dQ/dP their density ratio. Let N denote the number of samples simulated by the algorithm before it terminates. Then, for α ∈ (0 , 1) we have

where D α [ Q ‖ P ] = 1 α -1 log 2 E Z ∼ Q [ r ( Z ) α -1 ] is the Rényi α -divergence of Q from P .

GPRS induces a channel simulation protocol: We use an analogous solution to the standard rejection sampling-based scheme in Section 2.2, to obtain a channel simulation protocol for a pair of correlated random variables x , y ∼ P x , y . First, we assume that both the encoder and the decoder have access to the same realization of a (1 , P x ) -Poisson process Π , which they will use as their shared randomness. In a single round of communication, the encoder receives y ∼ P y , sets Q ← P x | y as the target and P ← P x as the proposal distribution, and runs Algorithm 3 to find the index N of the accepted arrival in Π . Finally, following Li & El Gamal (2018), they use a zeta distribution ζ ( n | s ) ∝ n -s with s -1 = I [ x ; y ] + 2 log 2 e to encode N using entropy coding. Then, the decoder simulates a P x | y -distributed sample by looking up the spatial coordinate X N of Π 's N th arrival. The following theorem shows that this protocol is optimally efficient; see Appendix B.4 for the proof.

Theorem 3.3 (Expected Codelength) . Let P x , y be a joint distribution over correlated random variables x and y , and let Π be the (1 , P x ) -Poisson process used by Algorithm 3. Then, the above protocol induced by the the algorithm is optimally efficient up to a constant, in the sense that

Note, that this result shows we can use GPRS as an alternative to the Poisson functional representation to prove of the strong functional representation lemma (Theorem 1; Li & El Gamal, 2018).

GPRS in practice: The real-world applicability of Algorithm 3 is limited by the following three computational obstacles: (1) GPRS requires exact knowledge of the density ratio dQ/dP to evaluate ϕ , which immediately excludes a many practical problems where dQ/dP is known only up to a normalizing constant. (2) To compute σ , we need to evaluate Equation (4) or (5), which requires us to compute w P and w Q , which is usually intractable for general dQ/dP . Fortunately, in some special cases they can be computed analytically: in Appendix G, we provide the expressions for discrete, uniform, triangular, Gaussian, and Laplace distributions. (3) However, even when we can compute w P and w Q , analytically evaluating the integral in Equation (5) is usually not possible. Moreover, solving it numerically is unstable as σ is unbounded. Instead, we numerically solve for σ -1 using Equation (4) in practice, which fortunately turns out to be stable.

However, we emphasize that GPRS's raison d'être is to perform channel simulation using the protocol we outline above or the ones we present in Section 3.1, not general-purpose random variate simulation. Practically relevant channel simulation problems, such as encoding a sample from the latent posterior of a VAE (Flamich et al., 2020) or encoding variational implicit neural representations (Guo et al., 2023; He et al., 2023) usually involve simple distributions such as Gaussians. In these cases, we have readily available formulae for w P and w Q and can cheaply evaluate σ -1 via numerical integration.

GPRS as greedy search: The defining feature of GPRS is that its acceptance criterion at each step is local , since if at step n the arrival ( T n , X n ) in Π falls under ϕ , the algorithm accepts it. Thus it is greedy search procedure. This is in contrast with A ∗ sampling (Maddison et al., 2014), whose acceptance criterion is global , as the acceptance of the n th arrival ( T n , X n ) depends on all other points of Π in the general case. In other words, using the language of Liu & Verdu (2018), GPRS is a causal sampler, as its acceptance criterion at each step only depends on the samples the algorithm has examined in previous steps. On the other hand, A* sampling is non-causal as its acceptance criterion in each step depends on future samples as well. Surprisingly, this difference between the search criteria does not make a difference in the average runtimes and codelengths in the general case. However, as we show in the next section, GPRS can be much faster in special cases.

3.1 Speeding up the greedy search

This section discusses two ways to improve the runtime of GPRS. First, we show how we can utilize available parallel computing power to speed up Algorithm 3. Second, we propose an advanced search

Algorithm 4: Parallel GPRS with J available threads.

Algorithm 5: Branch-and-bound GPRS R

strategy when the spatial domain Ω has more structure and obtain a super-exponential improvement in the runtime from exp 2 ( D ∞ [ Q ‖ P ]) to O ( D KL [ Q ‖ P ]) in certain cases.

Parallel GPRS: The basis for parallelizing GPRS is the superposition theorem (Kingman, 1992): For a positive integer J , let Π 1 , . . . Π J all be (1 /J, P ) -Poisson processes; then, Π = ⋃ J j =1 Π j is a (1 , P ) -Poisson process. As shown in Algorithm 4, this result makes parallelizing GPRS very simple given J parallel threads: First, we independently look for the first arrivals of Π 1 , . . . , Π J under the graph of ϕ , yielding ( T (1) N 1 , X (1) N 1 ) , . . . , ( T ( J ) N J , X ( J ) N J ) , respectively, where the N j corresponds to the index of Π j 's first arrival under ϕ . Then, we select the candidate with the earliest arrival time, i.e. j ∗ = arg min j ∈{ 1 ,...,J } T ( j ) N j . Now, by the superposition theorem, ( T ( j ∗ ) N j ∗ , X ( j ∗ ) N j ∗ ) is the first arrival of Π under ϕ , and hence X ( j ∗ ) N j ∗ ∼ Q . Finally, Alice encodes the sample via the tuple ( j ∗ , N j ∗ ) , i.e. which of the J processes the first arrival occurred in, and the index of the arrival in Π j ∗ . See the middle graphic in Figure 1 for an example with J = 2 parallel threads.

Our next results show that parallelizing GPRS results in a linear reduction in both the expectation and variance of its runtime and a more favourable codelength guarantee for channel simulation.

Theorem 3.4 (Expected runtime of parallelized GPRS) . Let Q,P and r be defined as above, and let ν j denote the random variable corresponding to the number of samples simulated by thread j in Algorithm 4 using J threads. Then, for all j ,

Theorem 3.5 (Expected codelength of parallelized GPRS) . Let P x , y be a joint distribution over correlated random variables x and y , and let Π 1 , . . . , Π J be (1 /J, P x ) -Poisson processes. Then, assuming log 2 J ≤ I [ x ; y ] , parallelized GPRS induces a channel simulation protocol such that

See Appendix C for the proofs. Note that we can use the same parallelisation argument with the appropriate modifications to speed up A ∗ sampling / coding too.

Branch-and-bound GPRS on R : We briefly restrict our attention to problems on R when the density ratio r is unimodal, as we can exploit this additional structure to more efficiently search for Π 's first

arrival under ϕ . Consider the example in the right graphic in Figure 1: we simulate the first arrival ( T 1 , X 1 ) , and reject it, since T 1 > ϕ ( X 1 ) . Since r is unimodal by assumption and σ is increasing, ϕ is also unimodal. Hence, for x < X 1 we must have ϕ ( x ) < ϕ ( X 1 ) , while the arrival time of any of the later arrivals will be larger than T 1 . Therefore, none of the arrivals to the left of X 1 will fall under ϕ either! Hence, it is sufficient to simulate Π to the right of X 1 , which we can easily do using generalized inverse transform sampling. 1 We repeat this argument for all further arrivals to obtain an efficient search procedure for finding Π 's first arrival under ϕ , described in Algorithm 5: We simulate the next arrival ( T B , X B ) of Π within some bounds B , starting with B = Ω . If T B ≤ ϕ ( X B ) we accept; otherwise, we truncate the bound to B ← B ∩ ( -∞ , X B ) , or B ← B ∩ ( X B , ∞ ) based on where X B falls relative to ϕ 's mode. We repeat these two steps until we find the first arrival.

Since Algorithm 5 does not simulate every point of Π , we cannot use the index N of the accepted arrival to obtain a channel simulation protocol as before. Instead, we encode the search path , i.e. whether we chose the left or the right side of our current sample at each step. Similarly to A* coding, we encode the path using its heap index (Flamich et al., 2022): the root has index H root = 1 , and for a node with index H , its left child is assigned index 2 H and its right child 2 H +1 . As the following theorems show, this version of GPRS is, in fact, optimally efficient; see Appendix E for proofs.

Theorem 3.6 (Expected Runtime of GPRS with binary search) . Let Q,P and r be defined as above, and let D denote the number of samples simulated by Algorithm 5 and λ = 1 / log 2 (4 / 3) Then,

Theorem 3.7 (Expected Codelength of GPRS with binary search) . Let P x , y be a joint distribution over correlated random variables x and y , and let Π be a (1 , P x ) -Poisson process. Then, GPRS with binary search induces a channel simulation protocol such that

Branch-and-bound GPRS with splitting functions: With some additional machinery, Algorithm 5 can be extended to more general settings, such as R D and cases where r is not unimodal, by introducing the notion of a splitting function . For a region of space, B ⊆ Ω , a splitting function split simply returns a binary partition of the set, i.e. { L, R } = split ( B ) , such that L ∩ R = ∅ and L ∪ R = B . In this case, we can perform a similar tree search to Algorithm 5, captured in Algorithm 6. Recall that ( ˜ T, ˜ X ) is the first arrival of Π under ϕ . Starting with the whole space B = Ω , we simulate the next arrival ( T, X ) of Π in B at each step. If we reject it, we partition B into two parts, L and R , using split . Then, with probability P [ ˜ X ∈ R | ˜ X ∈ B, ˜ T ≥ T ] , we continue searching through the arrivals of Π only in R , and only in L otherwise. We show the correctness of this procedure in Appendix F and describe how to compute P [ ˜ X ∈ R | ˜ X ∈ B, ˜ T ≥ T ] and σ in practice. This splitting procedure is analogous to the general version of A* sampling/coding (Maddison et al., 2014; Flamich et al., 2022), which is parameterized by the same splitting functions. Note that Algorithm 5 is a special case of Algorithm 6, where Ω = R , r is unimodal, and at each step for an interval bound B = ( a, b ) and sample X ∈ ( a, b ) we split B into { ( a, X ) , ( X,b ) } .

4 Experiments

We compare the average and one-shot case efficiency of our proposed variants of GPRS and a couple of other channel simulation protocols in Figure 2. See the figure's caption and Appendix H for details of our experimental setup. The top two plots in Figure 2 demonstrate that our methods' expected runtimes and codelengths align well with our theorems' predictions and compare favourably to other methods. Furthermore, we find that the mean performance is a robust statistic for the binary search-based variants of GPRS in that it lines up closely with the median, and the interquartile range is quite narrow. However, we also see that Theorem 3.7 is slightly loose empirically. We conjecture that the coefficient of the log 2 ( I [ x ; y ] + 1) term in Equation (12) could be reduced to 1 , and that the constant term could be improved as well.

On the other hand, the bottom plot in Figure 2 demonstrates the most salient property of the binary search variant of GPRS: unlike all previous methods, its runtime scales with D KL [ Q ‖ P ] and not D ∞ [ Q ‖ P ] . Thus, we can apply it to a larger family of channel simulation problems, where D KL [ Q ‖ P ] is finite but D ∞ [ Q ‖ P ] is large or even infinite, and other methods would not terminate.

Algorithm 6: Branch-and-bound GPRS with splitting function.

Input

:

Proposal distribution

P

,

Density ratio

r

=

dQ/dP

,

Stretch function

σ

,

Splitting function

split

.

Output:

Sample

X

∼

Q

and its

heap index

H

T

0

, H, B

(0

,

1

,

R

)

for

d

= 1

,

2

, . . .

do

X

d

∼

P

|

B

∆

d

∼

Exp(

P

(

B

))

T

d

T

d

-

1

+∆

d

if

T

d

< σ

(

r

(

X

d

))

then

return

X

d

, H

else

B

0

, B

1

split

(

B

)

ρ

P

X

∈

B

1

|

˜

X

∈

B,

˜

T

T

d

]

β

ρ

)

H

2

H

+

β

B

B

β

end

end

0

10

20

30

40

# of steps

GPRS

PFR

AS*

sGPRS

I

[

x

;

µ

]

UB

2

4

6

8

10

0

5

10

15

I

[

x

;

µ

]

Codelength

10

15

20

25

0

10

20

30

D

∞

[

Q

‖

P

]

# of steps

sGPRS

AS*

D

KL

D

∞

5 Related Work

Poisson processes for channel simulation: Poisson processes were introduced to the channel simulation literature by Li & El Gamal (2018) via their construction of the Poisson functional representation (PFR) in their proof of the strong functional representation lemma. Flamich et al. (2022) observed that the PFR construction is equivalent to a certain variant of A* sampling (Maddison et al., 2014; Maddison, 2016). Thus, they proposed an optimized version of the PFR called A* coding, which achieves O ( D ∞ [ Q ‖ P ]) runtime for one-dimensional unimodal distributions. GPRS was mainly inspired by A* coding, and they are dual constructions to each other in the following sense: Depth-limited A* coding can be thought of as an importance sampler , i.e. a Monte Carlo algorithm that returns an approximate sample in fixed time. On the other hand, GPRS is a rejection sampler , i.e. a Las Vegas algorithm that returns an exact sample in random time.

Channel simulation with dithered quantization: Dithered quantization (DQ; Ziv, 1985) is an alternative to rejection and importance sampling-based approaches to channel simulation. DQ exploits that for any c ∈ R and U, U ′ ∼ Unif( -1 / 2 , 1 / 2) , the quantities glyph[floorleft] c -U glyph[ceilingright] + U and c + U ′ are equal in distribution. While DQ has been around for decades as a tool to model and analyze quantization error, Agustsson & Theis (2020) reinterpreted it as a channel simulation protocol and used it to develop a V AE-based neural image compression algorithm. Unfortunately, basic DQ only allows uniform target distributions, limiting its applicability. As a partial remedy, Theis & Yosri (2022) showed DQ could be combined with other channel simulation protocols to speed them up and thus called their approach hybrid coding (HQ). Originally, HQ required that the target distribution be compactly supported, which was lifted by Flamich & Theis (2023), who developed an adaptive rejection sampler using HQ. In a different vein, Hegazy & Li (2022) generalize and analyze a method proposed in the appendix of Agustsson & Theis (2020) and show that DQ can be used to realize channel simulation protocols for one-dimensional symmetric, unimodal distributions.

Greedy Rejection Coding: Concurrently to our work, Flamich et al. (2023) introduce greedy rejection coding (GRC) which generalizes Harsha et al.'s rejection sampling algorithm to arbitrary probability spaces and arbitrary splitting functions, extending the work of Flamich & Theis (2023). Furthermore, they also utilize a space-partitioning procedure to speed up the convergence of their sampler and prove that a variant of their sampler also achieves optimal runtime for one-dimensional problems where dQ/dP is unimodal. However, the construction of their method differs significantly from ours. GRC is a direct generalization of Harsha et al. (2007)'s algorithm and, thus, a more 'conventional' rejection sampler, while we base our construction on Poisson processes. Thus, our proof techniques are also significantly different. It is an interesting research question whether GRC could be formulated using Poisson processes, akin to standard rejection sampling in Algorithm 2, as this could be used to connect the two algorithms and improve both.

6 Discussion and Future Work

Using the theory of Poisson processes, we constructed greedy Poisson rejection sampling. We proved the correctness of the algorithm and analyzed its runtime, and showed that it could be used to obtain a channel simulation protocol. We then developed several variations on it, analyzed their runtimes, and showed that they could all be used to obtain channel simulation protocols. As the most significant result of the paper, we showed that using the binary search variant of GPRS we can achieve O ( D KL [ Q ‖ P ]) runtime for arbitrary one-dimensional, unimodal density ratios, significantly improving upon the previous best O ( D ∞ [ Q ‖ P ]) bound by A* coding.

There are several interesting directions for future work. From a practical perspective, the most pressing question is whether efficient channel simulation algorithms exist for multivariate problems; finding an efficient channel simulation protocol for multivariate Gaussians would already have farreaching practical consequences. We also highlight an interesting theoretical question. We found that the most general version of GPRS needs to simulate exactly exp 2 ( D ∞ [ Q ‖ P ]) samples in expectation, which matches A* sampling. From Agustsson & Theis (2020), we know that for general problems, the average number of simulated samples has to be at least on the order of exp 2 ( D KL [ Q ‖ P ]) . Hence we ask whether this lower bound can be tightened to match the expected runtime of GPRS and A* sampling or whether there exists an algorithm that achieves a lower runtime.

References

Eirikur Agustsson and Lucas Theis. Universally quantized neural compression. In Advances in Neural Information Processing Systems , volume 33, 2020.

Harry Bateman and Arthur Erdélyi. Higher transcendental functions , volume 1. McGraw-Hill book company, 1953.

Robert M. Corless, Gaston H. Gonnet, David E. G. Hare, David J. Jeffrey, and Donald E. Knuth. On the Lambert W function. Advances in Computational mathematics , 5:329-359, 1996.

Gergely Flamich and Lucas Theis. Adaptive greedy rejection sampling. In IEEE International Symposium on Information Theory (ISIT) , pp. 454-459, 2023.

Gergely Flamich, Marton Havasi, and José Miguel Hernández Lobato. Compressing images by encoding their latent representations with relative entropy coding. Advances in Neural Information Processing Systems , 33, 2020.

Gergely Flamich, Stratis Markou, and José Miguel Hernández Lobato. Fast relative entropy coding with A* coding. In International Conference on Machine Learning , pp. 6548-6577, 2022.

Gergely Flamich, Stratis Markou, and José Miguel Hernández Lobato. Faster relative entropy coding with greedy rejection coding. In Advances in Neural Information Processing Systems , 2023.

Zongyu Guo, Gergely Flamich, Jiajun He, Zhibo Chen, and José Miguel Hernández Lobato. Compression with Bayesian implicit neural representations. In Advances in Neural Information Processing Systems , 2023.

Prahladh Harsha, Rahul Jain, David McAllester, and Jaikumar Radhakrishnan. The communication complexity of correlation. In Twenty-Second Annual IEEE Conference on Computational Complexity (CCC'07) , pp. 10-23. IEEE, 2007.

Marton Havasi, Robert Peharz, and José Miguel Hernández Lobato. Minimal random code learning: Getting bits back from compressed model parameters. In International Conference on Learning Representations , 2018.

Jiajun He, Gergely Flamich, Zongyu Guo, and José Miguel Hernández Lobato. RECOMBINER: Robust and enhanced compression with Bayesian implicit neural representations. arXiv preprint arXiv:2309.17182 , 2023.

Mahmoud Hegazy and Cheuk Ting Li. Randomized quantization with exact error distribution. In 2022 IEEE Information Theory Workshop (ITW) , pp. 350-355. IEEE, 2022.

Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations , 2014.

John F. C. Kingman. Poisson Processes . Oxford Studies in Probability. Clarendon Press, 1992. ISBN 9780191591242.

Cheuk Ting Li and Abbas El Gamal. Strong functional representation lemma and applications to coding theorems. IEEE Transactions on Information Theory , 64(11):6967-6978, 2018.

Jingbo Liu and Sergio Verdu. Rejection sampling and noncausal sampling under moment constraints. In 2018 IEEE International Symposium on Information Theory (ISIT) , pp. 1565-1569. IEEE, 2018.

Chris J. Maddison. A Poisson process model for Monte Carlo. Perturbation, Optimization, and Statistics , pp. 193-232, 2016.

Chris J. Maddison, Daniel Tarlow, and Tom Minka. A* sampling. In Advances in Neural Information Processing Systems , volume 27, pp. 3086-3094, 2014.

Pat Muldowney, Krzysztof Ostaszewski, and Wojciech Wojdowski. The Darth Vader rule. Tatra Mountains Mathematical Publications , 52(1):53-63, 2012.

Jorma Rissanen and Glen G Langdon. Arithmetic coding. IBM Journal of research and development , 23(2):149-162, 1979.

Abhin Shah, Wei-Ning Chen, Johannes Balle, Peter Kairouz, and Lucas Theis. Optimal compression of locally differentially private mechanisms. In International Conference on Artificial Intelligence and Statistics , pp. 7680-7723. PMLR, 2022.

Lucas Theis and Noureldin Yosri. Algorithms for the communication of samples. In International Conference on Machine Learning , 2022.

Lucas Theis, Tim Salimans, Matthew D. Hoffman, and Fabian Mentzer. Lossy compression with Gaussian diffusion. arXiv:2206.08889, 2022.

Jacob Ziv. On universal quantization. IEEE Transactions on Information Theory , 31(3):344-347, 1985.

A Measure-theoretic Construction of Greedy Poisson Rejection Sampling

In this section, we provide a construction of greedy Poisson rejection sampling (GPRS) in its greatest generality. However, we first establish the notation for the rest of the appendix.

Notation: In what follows, we will always denote GPRS's target distribution as Q and its proposal distribution as P . We assume that Q and P are Borel probability measures over some Polish space Ω with Radon-Nikodym derivative r def = dQ/dP . We denote the standard Lebesgue measure on R n by λ . All logarithms are assumed to be to the base 2 denoted by log 2 . Similarly, we use the less common notation of exp 2 to denote exponentiation with base 2 , i.e. exp 2 ( x ) = 2 x . The relative entropy of Q from P is defined as D KL [ Q ‖ P ] def = ∫ Ω log 2 r ( x ) dP ( x ) and the Rényi ∞ -divergence as D ∞ [ Q ‖ P ] def = ess sup x ∈ Ω { log r ( x ) } , where the essential supremum is taken with respect to P .

Restricting a Poisson process 'under the graph of a function': We first consider the class of functions 'under which' we will be restricting our Poisson processes. Let ϕ : Ω → R + be a measurable function, such that | ϕ (Ω) | = | r (Ω) | , i.e. the image of ϕ has the same cardinality as the image of r = dQ/dP . This implies that there exist bijections between r (Ω) and ϕ (Ω) . Since both images are subsets of R + , both images have a linear order. Hence, a monotonically increasing bijection ˜ σ exists such that ϕ = ˜ σ ◦ r . Since ˜ σ is monotonically increasing, we can extend it to a monotonically increasing continuous function σ : R + → R + . This fact is significant, as it allows us to reduce questions about functions of interest from Ω to R + to invertible functions on the positive reals. Since σ is continuous and monotonically increasing on the positive reals, it is invertible on the extended domain. We now consider restricting a Poisson process under the graph of ϕ , and work out the distribution of the spatial coordinate of the first arrival.

Let Π = { ( T n , X n ) } ∞ n =1 be a Poisson process over R + × Ω with mean measure µ = λ × P . Let U def = { ( t, x ) ∈ Ω | t ≤ ϕ ( x ) } be the set of points 'under the graph' of ϕ . By the restriction theorem (Kingman, 1992), ˜ Π def = Π ∩ U is a Poisson process with mean measure ˜ µ ( A ) = µ ( A ∩ U ) for an arbitrary Borel set A . As a slight abuse of notation, we define the shorthand ˜ µ ( t ) = ˜ µ ([0 , t ) × Ω) for the average number of points in ˜ Π up to time t . Let N ˜ Π denote the counting measure of ˜ Π and let ( ˜ T, ˜ X ) be the first arrival of ˜ Π , i.e. the first point of the original process Π that falls under the graph of ϕ , assuming that it exists . To develop GPRS, we first work out the distribution of ˜ X for an arbitrary ϕ .

Remark: Note, that the construction below only holds if the first arrival is guaranteed to exist, otherwise the quantities below do not make sense. From a formal point of view, we would have to always condition on the event N ˜ Π ( ∞ ) def = lim t →∞ N ˜ Π ( t ) > 0 , e.g. we would need to write P [ ˜ T ≥ t | N ˜ Π ( ∞ ) > 0] . However, we make a 'leap of faith' instead and assume that our construction is valid to obtain our 'guesses' for the functions σ and ϕ . Then, we show that for our specific guesses the derivation below is well-defined, i.e. namely that P [ N ˜ Π ( ∞ ) > 0] = 1 , which then implies P [ ˜ T ≥ t | N ˜ Π ( ∞ ) > 0] = P [ ˜ T ≥ t ] . To do this, note that

Hence, for the well-definition of our construction it is enough to show that for our choice of σ , ˜ µ ( t ) →∞ as t →∞ .

Constructing σ : We wish to derive

To do this, recall that we defined N ˜ Π ( t ) def = N ˜ Π ([0 , t ) × Ω) and define N ˜ Π ( s, t ) def = N ˜ Π ([ s, t ] × Ω) . Similarly, let ˜ µ ( t ) def = E [ N ˜ Π ( t )] and ˜ µ ( s, t ) def = E [ N ˜ Π ( s, t )] = ˜ µ ( t ) -˜ µ ( s ) . Then,

where the last equality holds, because N ˜ Π ( s ) ⊥ N ˜ Π ( s, t ) , since [0 , s ) × Ω and [ s, t ] × Ω are disjoint. By Lemma 3 from Maddison (2016),

Substituting this back into Equation (18) and using the fact that N ˜ Π is Poisson, we find

where the last equation holds by the definition of the derivative and the fundamental theorem of calculus. From this, by inspection we find

Finally, by marginalizing out the first arrival time, we find that the spatial distribution is

Deriving ϕ by finding σ : Note that Equation (27) holds for any ϕ . However, to get a correct algorithm, we need to set it such that d P [ ˜ X = x ] dP = dQ dP = r . Since we can write ϕ = σ ◦ r by our earlier argument, this problem is reduced to finding an appropriate invertible continuous function σ : R + → R + . Thus, we wish to find σ such that

Now, introduce τ = σ ( r ( x )) , so that σ -1 ( τ ) = r ( x ) . Note that this substitution only makes sense for τ ∈ r (Ω) . However, since we are free to extend σ -1 to R + in any we like so long as it is monotone, we may require that this equation hold for all τ ∈ R + . Then, we find

with σ -1 (0) = 0 . Before we solve for σ , we define

Note that w P and w Q are supported on [0 , r ∗ ) , where r ∗ def = ess sup x ∈ Ω { r ( x ) } = exp 2 ( D ∞ [ Q ‖ P ]) . Now, we can rewrite Equation (30) as

where the second term in the last equation follows by noting that

Now, by the inverse function theorem, we get

Thus, to recapitulate, setting ϕ = σ ◦ r , where σ is given by Equation (42) will ensure that the spatial distribution of the first arrival of Π under ϕ is the target distribution Q .

An alternate form for P [ ˜ T ≥ t ] : We make use of the following integral representation of the indicator function to present an alternative form of the survival function of ˜ T :

where δ is the Dirac delta function. Then, we have

thus we finally find

Then, from this identity, we find that

Well-definition of GPRS and Algorithm 3 terminates with probability 1 : To ensure that our construction is useful, we need to show that the first arrival under the graph ˜ T exists and is almost surely finite, so that Algorithm 3 terminates with probability 1 . First, note that for any t > 0 , we have ˜ µ ( t ) ≤ µ ( t ) = t < ∞ . Since P [ N ˜ Π ( t ) = 0] = e -˜ µ ( t ) , this implies that for any finite time t , P [ N ˜ Π ( t ) = 0] > 0 . Second, note that for h ∈ [0 , r ∗ ] ,

from which

Thus, in particular, we find that

By continuity, this can only hold if ˜ µ ( σ ( h )) →∞ as h → r ∗ . Since ˜ µ ( t ) is finite for all t > 0 and σ is monotonically increasing, this also implies that σ ( h ) →∞ as h → r ∗ . Thus, we have shown a couple of facts:

σ ( h ) → ∞ as h → r ∗ , hence ϕ is always unbounded at points in Ω that achieve the supremum r ∗ . Furthermore, this implies that

Since σ -1 is increasing, this implies that it is bounded from above by r ∗ .

˜ µ ( t ) < ∞ for all t > 0 , but ˜ µ ( t ) → ∞ as t → ∞ . Hence, by Equation (14) we have P [ N ˜ Π ( ∞ ) > 0] = 1 . Thus, the first arrival of ˜ Π exists almost surely, and our construction is well-defined. In particular, it is meaningful to write P [ ˜ T ≥ t ] .

P [ ˜ T ≥ t ] = P [ N ˜ Π ( t ) = 0] → 0 as t →∞ , which shows that the first arrival time is finite with probability 1 . In turn, this implies that Algorithm 3 will terminate with probability one, as desired.

Note, that w P and w Q can be computed in many practically relevant cases, see Appendix G.

B Analysis of Greedy Poisson Rejection Sampling

Now that we constructed a correct sampling algorithm in Appendix A, we turn our attention to deriving the expected first arrival time ˜ T in the restricted process ˜ Π , the expected number of samples N before Algorithm 3 terminates and bound on N 's variance, and an upper bound on its coding cost. The proofs below showcase the true advantage of formulating the sampling algorithm using the language of Poisson processes: the proofs are all quite short and elegant.

B.1 The Expected First Arrival Time

At the end of the previous section, we showed that ˜ T is finite with probability one, which implies that E [ ˜ T ] < ∞ . Now, we derive the value of this expectation exactly. Since ˜ T is a positive random variable, by the Darth Vader rule (Muldowney et al., 2012), we may write its expectation as

B.2 The Expectation and Variance of the Runtime

Expectation of N : First, let ˜ Π C def = Π \ ˜ Π be the set of points in Π above the graph of ϕ . Since ˜ Π and ˜ Π C are defined on complementary sets, they are independent. Since by definition ˜ T is the first arrival of ˜ Π and the N th arrival of Π , it must mean that the first N -1 arrivals of Π occurred in ˜ Π C . Thus, conditioned on ˜ T , we have N -1 = |{ ( T, X ) ∈ ˜ Π C | T < ˜ T }| is Poisson distributed with mean

By the law of iterated expectations,

Focusing on the second term, we find

where the third equality follows by substituting u = ˜ µ ( t ) . Finally, plugging the above and Equation (57) into Equation (62), we find

Variance of N : We now show that the distribution of N is super-Poissonian , i.e. E [ N ] ≤ V [ N ] . Similarly to the above, we begin with the law of iterated variances to find

where the second equality follows from the fact that the variance of N -1 matches its mean conditioned on ˜ T , since it is a Poisson random variable. Focussing on the last term, we find

where the third equality follows from a similar u -substitution as in Equation (65). Thus, putting everything together, we find

B.3 The Fractional Moments of the Index

In this section, we follow the work of Liu & Verdu (2018) and analyse the fractional moments of the index N and prove Theorem 3.2. As before, set Q and P as GPRS's target and proposal distribution, respectively, and let r = dQ dP . To begin, we define the Rényi α -divergence between two distributions for α ∈ (0 , 1) ∪ (1 , ∞ ) as

By taking limits, we can extend the definiton to α = 1 , ∞ as

Now fix α ∈ (0 , 1) and x ∈ Ω and note that by Jensen's inequality,

Now, to continue, we will show the following bound:

Let Π be the (1 , P ) -Poisson process realized by Algorithm 3, and let

i.e. the index of the first arrival in Π such that the density ratio of the spatial coordinate exceeds r ( x ) . Since X i ∼ P are i.i.d.,

Now, since geometrically distributed random variables are memoryless, we find

Before we proceed, we first require the following result

Lemma B.1. Let all quantities be as defined above and let x, y ∈ Ω such that r ( x ) ≤ r ( y ) . Then, N | ˜ X = y stochastically dominates N | ˜ X = x , i.e. for all positive integers n we have

Proof. The proof follows via direct calculation. Concretely,

where Γ( s, x ) is the upper incomplete Γ -function defined as

hence which then implies

Equation (87) follows from the fact that conditioned on ˜ T we have that N -1 is a Poisson random variable with mean ˜ T -˜ µ ( ˜ T ) . Now, focussing on the expectation term in Equation (87), we let

Substituting h = r ( x ) , we now show that glyph[epsilon1] is a decreasing function, which proves the lemma. To this end, regardless of the image of the density ratio r (Ω) , extend the definition of glyph[epsilon1] to the entirety of (0 , ess sup r ( x )) . Now, observe, that

Focussing on the integral, via integration by parts we find

Plugging this back into the above calculation, we get

Note, that since the integrand is positive and h was arbitrary, dglyph[epsilon1] dh is negative for every h in its domain, which proves the lemma.

Returning to Equation (83), we now examine the two cases M < N and M = N separately.

Case M < N : In this case, we have the following inequalities:

r ( x ) ≤ r ( X M ) by the definition of M . Applying σ to both sides we find ϕ ( x ) ≤ ϕ ( X M ) .

ϕ ( X M ) < T M , since M < N and by definition N is the index of the first arrival of Π under the graph of ϕ .

Now, note that since M < N conditioned on T M , the first M -1 points of Π are all rejected and hence are all in ˜ Π C = Π \ ˜ Π . Thus, M -1 | ( M < N,T M ) is Poisson distributed with mean T M -˜ µ ( T M ) . Hence,

On the other hand, since the N -th point is the index of first arrival of Π under the graph of ϕ , hence ˜ T ≤ ϕ ( ˜ X ) . Furthermore, conditioning on ˜ X = x we get ˜ T ≤ ϕ ( x ) . Similarly to the above case N -1 | ˜ T is Poisson distributed with mean ˜ T -˜ µ ( ˜ T ) . Hence, we get

Putting the above two inequalities, we find

Case M = N : In this case, we have

where Equation (105) follows by applying Lemma B.1. Finally, putting these two cases together we find

which establishes Equation (77). Now, taking expectation over ˜ X , we get

In the above, Equation (113) follows by integration by parts and from the fact that

We now note that we can bound w P using Markov's inequality:

Putting the above in Equation (114), we get

B.4 The Codelength of the Index

As discussed in Section 3, Alice and Bob can realize a one-shot channel simulation protocol using GPRS for a pair of correlated random variables x , y ∼ P x , y if they have access to shared randomness and can simulate samples from P x . In particular, after receiving y ∼ P y , Alice runs GPRS with proposal distribution P x and target P x | y , and the shared randomness to simulate the Poisson process Π . She then encodes the index N of the first arrival of Π under the graph of ϕ . Bob can decode Alice's sample x ∼ P x | y by simulating the same N samples from Π using the shared randomness. The question is, how efficiently can Alice encode N ? We answer this question by following the approach of Li & El Gamal (2018). Namely, we first bound the conditional expectation E [log 2 N | y = y ] , after which we average over y to bound E [log 2 N ] . Then, we use the maximum entropy distribution subject to the constraint of fixed E [log 2 N ] to bound H [ N ] . Finally, noting that x is a function of Π and N , we get

from which which will finish the proof.

Bound on the conditional expectation: Fix y and set Q = P x | y and P = P x as GPRS's target and proposal distribution, respectively. Let ( t, x ) ∈ R + × Ω be a point under the graph of ϕ , i.e. σ -1 ( t ) ≤ r ( x ) . Then,

Next, conditioning on the first arrival ( ˜ T, ˜ X ) we get

From this, we get

where the first inequality follows by Jensen's inequality.

Now, by the law of iterated expectations, we find

where the second inequality follows from switching to the natural base and using a first-order Taylor approximation for e 1 x .

Bound on the marginal expectation and entropy: Equation (143) is a one-shot bound, which yields

Taking expectation over y ∼ P y , we get

Finally, following Proposition 4 in Li & El Gamal (2018), the maximum entropy distribution for N subject to a constraint on E [log 2 N ] obeys

Plugging in Equation (145) into the above, we get

Similarly to Li & El Gamal (2018), we can encode N using a Zeta distribution ζ ( n | s ) ∝ n -s with

With this choice of the coding distribution, the expected codelength of N is upper bounded by I [ x ; y ] + log 2 ( I [ x ; y ] + 1) + 7 bits.

C Analysis of Parallel GPRS

Parallel GPRS (PGPRS) is a general technique to accelerate GPRS when parallel computing power is available and is presented in Section 3.1. Assuming that J parallel threads are available, for target distribution Q and proposal P , PGPRS simulates J Poisson processes Π 1 , . . . , Π J in parallel, all of them with mean measure µ i def = λ J × P . Assuming {( T (1) N 1 , X (1) N 1 ) , . . . , ( T ( J ) N J , X ( J ) N J )} are the first arrivals of Π 1 , . . . , Π J under ϕ , respectively, PGPRS selects the arrival with the overall smallest arrival time J ∗ = arg min j ∈{ 1 ,...,J } { T ( j ) N j } . By the superposition theorem (Kingman,

1992), ∪ J j =1 Π j = Π is a Poisson process with mean measure µ = λ × P , hence ( T ( J ∗ ) N J ∗ , X ( J ∗ ) N J ∗ ) will be the first arrival of Π under the graph of ϕ and the theoretic construction in Appendix A therefore guarantees the correctness of PGPRS.

Expected runtime: We will proceed similarly to the analysis in Appendix B.2. Concretely, let ˜ T = T ( J ∗ ) N J ∗ be the first arrival time of Π under the graph of ϕ . Let

be the number of points in Π j that are rejected before the global first arrival occurs. By an independence argument analogous to the one given in Appendix B.2, the ν 1 , . . . , ν J are independent Poisson random variables, each distributed with mean

Hence, by the law of iterated expectations, we find that we get

rejections in each of the J threads on average before the global first arrival occurs. Now, a thread j terminates either when the global first arrival occurs in it or when the thread's current arrival time T ( j ) n j provably exceeds the global first arrival time. Thus, on average, each thread will have one more rejection compared to Equation (154). Hence the average runtime of a thread is

and the average number of samples simulated by PGPRS across all its threads is

Variance of Runtime: Once again, similarly to Appendix B.2, we find by the law of iterated variances that the variance of the runtime in each of the j threads is

meaning that we make an O (1 /J ) reduction in the variance of the runtime compared to regular GPRS.

Codelength: PGPRS can also realize a one-shot channel simulation protocol for a pair of correlated random variables x , y ∼ P x , y . For a fixed y ∼ P y , Alice applies PGPRS to the target Q = P x | y and proposal P = P x , and encodes the two-part code ( J ∗ , N J ∗ ) . Bob can then simulate N J ∗ samples from Π J ∗ and recover Alice's sample.

Encoding J ∗ : By the symmetry of the setup, the global first arrival will occur with equal probability in each subprocess Π j . Hence J ∗ follows a uniform distribution on { 1 , . . . J } . Therefore, Alice can encode J ∗ optimally using glyph[ceilingleft] log 2 J glyph[ceilingright] bits.

Encoding N J ∗ : We can develop a bound using an almost identical argument to the one in Appendix B.4. In particular, by adapting the conditional bound in Equation (138) appropriately using Equation (155), we get

Then, using this conditional bound and adapting Equation (143), we find

to obtain a one-shot bound. Taking expectation over y ∼ P y , we get

hence, by adapting the maximum entropy bound in Equation (150), we find

Thus, we finally find that the entropy of the two-part code ( J ∗ , N J ∗ ) is upper bounded by

Using a Zeta distribution ζ ( n | s ) ∝ n -s to encode N J ∗ | J ∗ with

we find that the expected codelength of the two-part code is upper bounded by

D Simulating Poisson Processes Using Tree-Structured Partitions

glyph[negationslash]

glyph[negationslash]

In this section, we examine an advanced simulation technique for Poisson processes, which is required to formulate the binary search-based variants of GPRS. We first recapitulate the tree-based simulation technique from (Flamich et al., 2022) and some important results from their work. Then, we present Algorithm 7, using which we can finally formulate the optimally efficient variant of GPRS in Appendix E. Note: For simplicity, we present the ideas for Poisson processes whose spatial measure P is non-atomic. These ideas can also be extended to atomic spatial measures with appropriate modifications.

Splitting functions: Let Π be a spatiotemporal Poisson process on some space Ω with non-atomic spatial measure P . In this paragraph, we will not deal with Π itself yet, but the space on which it is defined and the measure P . Now, assume that there is a function split , which for any given Borel set B ⊆ Ω , produces a (possibly random) P -essential partition of B consisting of two Borel sets, i.e.

The last condition simply allows split to discard some points from B that will not be included in either the left or right split. For example, if we wish to design a splitting function for a subset of

B ⊆ R d to split B along some hyperplane, we do not need to include the points on the hyperplane in either set. The splitting function that always exists is the trivial splitting function, which just returns the original set and the empty set:

A more interesting example when Ω = R is the on-sample splitting function, where for a R -valued random variable X ∼ P | B it splits B as

This function is used by Algorithm 5 and AS* coding (Flamich et al., 2022). Another example is the dyadic splitting function operating on a bounded interval ( a, b ) , splitting as

We call this dyadic because when we apply this splitting function to (0 , 1) and subsets produced by applying it, we get the set of all intervals with dyadic endpoints on (0 , 1) , i.e. { (0 , 1) , (0 , 1 / 2) , (1 / 2 , 1) , (0 , 1 / 4) , . . . } . This splitting function is used by Algorithm 6 and AD* coding. As one might imagine, there might be many more possible splitting functions than the two examples we give above, all of which might be more or less useful in practice.

Split-induced binary space partitioning tree: Every splitting function on a space Ω induces an infinite set of subsets by repeatedly applying the splitting function to the splits it produces. These sets can be organised into an infinite binary space partitioning tree (BSP-tree) , where each node in the tree is represented by a set produced by split and an unique index. Concretely, let the root of the tree be represented by the whole space Ω and the index H root = 1 . Now we recursively construct the rest of the tree as follows: Let ( B,H ) be a node in the tree, with B a Borel set and H its index, and let { L, R } = split ( B ) be the left and right splits of B . Then, we set ( L, 2 H ) as the node's left child and ( R, 2 H +1) as its right child. We refer to each node's associated index as their heap index .

Heap-indexing the points in Π and a strict heap invariant: As we saw in Section 2.1, each point in Π can be uniquely identified by their time index N , i.e. if we time-order the points of Π , N represents the N th arrival in the ordered list. However, we can also uniquely index each point in Π using a splitting function-induced BSP-tree as follows.

We extend each node in the BSP-tree with a point from Π , such that the extended tree satisfies a strict heap invariant : First, we extend the root node (Ω , 1) by adjoining Π 's first arrival ( T 1 , X 1 ) and the first arrival index 1 to get ( T 1 , X 1 , Ω , 1 , 1) . Then, for every other extended node ( T, X, B, H, N ) with parent node ( T ′ , X ′ , B ′ , glyph[floorleft] H/ 2 glyph[floorright] , M ) we require that ( T, X ) is the first arrival in the restricted process Π ∩ (( T ′ , ∞ ) × B ) . This restriction enforces that we always have that T ′ < T and, therefore, M < N . Furthermore, it is strict because there are no other points of Π in B between those two arrival times.

Notation for the tree structure: Let us denote the extended BSP tree T on Π induced by split . Each node ν ∈ T is a 5-tuple ν = ( T, X, B, H, N ) consisting of the arrival time T , spatial coordinate X , its bounds B , split -induced heap index H and time index N . As a slight overload of notation, for a node ν , we use it as subscript to refer to its elements, e.g. T ν is the arrival time associated with node ν . We now prove the following important result, which ensures that we do not lose any information by using T to represent Π . An essentially equivalent statement was first proven for Gumbel processes (Appendix, 'Equivalence under partition ' subsection; Maddison et al., 2014).

Lemma D.1. Let Π , P , split and T be as defined above. Then, P -almost surely, T contains every point of Π .

Proof. We give an inductive argument.

Base case: the first arrival of Π is by definition contained in the root node of T .

Inductive hypothesis: assume that the first N arrivals of Π are all contained in T .

Case for N +1 : We begin by observing that if the first N nodes are contained in T , they must form a connected subtree T N . To see this, assume the contrary, i.e. that the first N arrivals form a subgraph Γ N ⊆ T with multiple disconnected components. Let ν ∈ Γ N be a node contained in a component

of Γ N that is disconnected from the component of Γ N containing the root node. Since T is a tree, there is a unique path π from the root node to ν in T , and since ν and the root are disconnected in Γ N , π must contain a node c glyph[negationslash]∈ Γ N . However, since c is an ancestor of ν , by the heap invariant of T we must have that the time index of c is N c < N ν ≤ N hence c ∈ Γ N , a contradiction.

Thus, let T N represent the subtree of T containing the first N arrivals of Π . Now, let F N represent the frontier of T N , i.e. the leaf nodes' children:

where parent retrieves the parent of a node in T . Let

be the union of all the bounds of the nodes in the frontier. A simple inductive argument shows that for all N the nodes in F N provide a P -essential partition of Ω , from which P ( ¯ Ω N ) = 1 . Let T N be the N th arrival time of Π . Now, by definition, the arrival time associated with every node in the frontier F N must be later than T N . Finally, consider the first arrival time across the nodes in the frontier:

Then, conditioned on T N , T ∗ N is the first arrival of Π restricted to ( T N , ∞ ) × ¯ Ω , thus it is P -almost surely the N +1 st arrival in Π , as desired.

A connection between the time index an heap index of a node: Now that we have two ways of uniquely identifying each point in Π it is natural to ask whether there is any relation between them. In the next paragraph we adapt an argument from Flamich et al. (2022) to show that under certain circumstances, the answer is yes.

First, we need to define two concepts, the depth of a node in an extended BSP-tree and a contractive splitting function. Thus, let T be an extended BSP-tree over Π induced by split . Let ν ∈ T . Then, the depth D of ν in T is defined as the distance from ν to the root. A simple argument shows that the heap index H ν of every node ν at depth D in T is between 2 D ≤ H ν < 2 D +1 . Thus, we can easily obtain the depth of a node ν from the heap index via the formula D ν = glyph[floorleft] log 2 H ν glyph[floorright] . Next, we say that split is contractive if all the bounds it produces shrink on average. More formally, let glyph[epsilon1] < 1 , and for a node ν ∈ T , let A ν denote the set of ancestor nodes of ν . Then, split is contractive if for every non-root node ν ∈ T we have

where T A ν and denotes the subtree of T containing the arrivals of the ancestors of ν .

Note that glyph[epsilon1] ≥ 1 / 2 for any split function. This is because if { L, R } = split ( B ) for some set B , then P ( R ) = P ( B ) -P ( L ) . Thus, if P ( R ) = αP ( B ) , then P ( L ) = (1 -α ) P ( B ) , and by definition glyph[epsilon1] = max { α, 1 -α } , which is minimized when α = 1 / 2 , from which glyph[epsilon1] = 1 / 2 .

For example, split dyad is contractive with glyph[epsilon1] = 1 / 2 , while split trivial is not contractive. By Lemma 1 from Flamich et al. (2022), split on-samp is also contractive with glyph[epsilon1] = 3 / 4 . We now strengthen this result using a simple argument, and show that split on-samp is contractive with glyph[epsilon1] = 1 / 2 .

Lemma D.2. Let ν, D be defined as above, let P be a non-atomic probability measure over R with CDF F P . Let Π a (1 , P ) -Poisson process and T be the BSP-tree over Π induced by split on-samp . Then,

Proof. Fix D ν = d , and let N d = { n ∈ T | D n = d } be the set of nodes in T whose depth is d . Note, that |N d | = 2 d . We will show that

i.e. that the distributions of the bound sizes are all the same. From this, we will immediately have E [ P ( B n )] = E [ P ( B ν )] for every n ∈ N d . Then, using the fact, that the nodes in N d for a P -almost partition of Ω , we get:

Dividing the very left and very right by |N d | = 2 d yields the desired result.

To complete the proof, we now show that by symmetry, Equation (174) holds. We begin by exposing the fundamental symmetry of split on-samp : for a node ν with left child L and right child R , the left and right bound sizes are equal in distribution:

First, note that by definition, all involved bounds will be intervals. Namely, assume that B ν = ( a, b ) for some a < b and X ν is the sample associated with ν . Then, B L = ( a, X ν ) and B R = ( X ν , b ) and hence P ( B L ) = F P ( X ν ) -F P ( a ) . Since X ν ∼ P | B ν , by the probability integral transform, we have F ( X ν ) ∼ Unif( F P ( a ) , F P ( b )) , from which P ( B L ) ∼ Unif(0 , F P ( b ) -F P ( a )) = Unif(0 , P ( B ν )) . Since P ( B R ) = P ( B ν ) -P ( B L ) , we similarly have P ( B R ) ∼ Unif(0 , P ( B ν )) , which establishes our claim.

Now, to show Equation (174), fix d and fix n ∈ N d . Let A n denote the ancestor nodes of n . As we saw in the paragraph above,

regardless of whether n is a left or a right child of its parent. We can apply this d times to the ancestors of n find the marginal:

Since the choice of n was arbitrary, all nodes in N d have this distribution, which is what we wanted to show.

Now, we have the following result.

Lemma D.3. Let split be a contractive splitting function for some glyph[epsilon1] ∈ [1 / 2 , 1) . Then, for every node ν in T with time index N ν and depth D ν , we have

Proof. Let us examine the case N ν = 1 first. In this case, ν is the root of T and has depth D ν = 0 by definition. Thus, 0 ≤ -log glyph[epsilon1] 1 holds trivially.

Now, fix ν ∈ T with time index N ν = N > 1 . Let T N -1 be the subtree of T containing the first N -1 arrivals, F N -1 be the frontier of T N -1 and T N -1 the ( N -1) st arrival time. Then, as we saw in Lemma D.1, the N th arrival in Π occurs in one of the nodes in the frontier F N -1 , after T N -1 . In particular, conditioned on T N -1 , the arrival times associated with each node f ∈ F N -1 will be shifted exponentials T f = T N -1 +Exp( P ( B f )) , and the N th arrival time in Π is the minimum of these: T ν = T N = min f ∈F N -1 T f . It is a standard fact (see e.g. Lemma 6 in Maddison (2016)) that the index of the minimum

is independent of T ν = T F N and P [ F N = f | T N -1 ] = P ( B f ) . A simple inductive argument shows that the number of nodes on the frontier |F N -1 | = N . Thus, we have a simple upper bound on the entropy of F :

Thus, taking expectation over T N -1 , we find

The second inequality holds by Jensen's inequality. In the third inequality, we apply Equation (172), and one might worry about conditioning on N ν here. However, this is not an issue because

Thus, we finally get

by dividing both sides by -log 2 glyph[epsilon1] and we obtain the desired result.

Converting between different heap indices: Assume now that we have two splitting functions, split target and split sim , which induce their own BSP-ordering on Π , T target and T sim. Now, given a split target -induced heap index H , Algorithm 7 presents a method for simulating the appropriate node ν ∈ T target by simulating nodes from T sim. In other words, given a node with some heap index induced by a splitting function, Algorithm 7 lets us find the heap index of the same arrival induced by a different splitting function. The significance of Algorithm 7 is that it lets us develop convenient search methods using a given splitting function, but it might be more efficient to encode the heap index induced by another splitting function.

Theorem D.4. Let Π , split target , split sim , T target and T sim be as above. Let ν ∈ T target with and let ( T sim , X sim , H sim ) be the arrival and its heap index output by Algorithm 7 given the above as input as well as H ν as the target index. Then, Algorithm 7 is correct, in the sense that

and H sim is the heap index of ( T sim , X sim ) in T sim .

Proof. First, observe that when split target = split sim , Algorithm 7 collapses onto just the extended BSP tree construction for Π and simply returns the arrival with the given heap index H target in T target . In particular, the inner loop will always exit after one iteration, and every time one and only one of the conditional blocks will be executed. In other words, in this case, the algorithm becomes equivalent to Algorithm 2 in Flamich et al. (2022).

glyph[negationslash]

Let us now consider the case when split target = split sim . Denote the depth of the required node by D ν = glyph[floorleft] log 2 H ν glyph[floorright] . Now, we give an inductive argument for correctness.

Base case: Consider D ν = 0 . In this case, the target bounds B = Ω , and the first sample we draw X ∼ P is guaranteed to fall in Ω . Hence, for D ν = 0 the outer loop only runs for one iteration. Furthermore, during that iteration, the inner loop will also exit after one iteration, and Algorithm 7 returns the sample ( T, X ) sampled before the outer loop with heap index 1 . Since T ∼ Exp(1) and X ∼ P , this will be a correctly distributed output with the appropriate heap index.

Inductive hypothesis: Assume Algorithm 7 is correct heap indices with depths up to D ν = d .

Case D ν = d +1 : Let ρ ∈ T target be the parent node of ν with arrival ( T ρ , X ρ ) . Then, D ρ = d , hence by the inductive hypothesis, Algorithm 7 will correctly simulate a branch T target up to node ρ . At the end of the d th iteration of the outer loop Algorithm 7 sets the target bounds B ← B ν . Then, in the final, d +1 st iteration, the inner loop simply realizes T sim and accepts the first sample after X that falls inside B ν whose time index T > T ρ . Due to the priority queue, the loop simulates the nodes of T sim in time order; hence the accepted sample will also be the one with the earliest arrival time. Furthermore, Algorithm 7 only ever considers nodes of T sim whose bounds intersect the target bounds B ν , hence the inner loop is guaranteed to terminate, which finishes the proof.

E GPRS with Binary Search

We now utilise the machinery we developed in Appendix D to analyze Algorithm 5.

Correcntess of Algorithm 5: Observe that Algorithm 5 constructs the extended BSP tree for the on-sample splitting function. Thus, we will now focus on performing a binary tree search using the extended BSP tree induced by the on-sample splitting function, which we denote by T . The first step of the algorithm matches GPRS's first step (Algorithm 3). Hence it is correct for the first step. Now consider the algorithm in an arbitrary step k before termination, where the candidate sample ( T, X ) is rejected, i.e. T > ϕ ( X ) . By assumption, the density ratio r is unimodal, and since σ is monotonically increasing, ϕ = σ ◦ r is unimodal too. Thus, let x ∗ ∈ Ω be such that r ( x ∗ ) = r ∗ , where r ∗ = exp 2 ( D ∞ [ Q ‖ P ]) . Assume for now that X < x ∗ , the case X > x ∗ follows by a symmetric argument. By the unimodality assumption, since X < x ∗ , it must hold that for all y < X , we have ϕ ( y ) < ϕ ( X ) . Consider now the arrival ( T L , X L ) of Π in the current node's left child. Then, we will have T < T L and X L < X by construction. Thus, finally, we get

meaning that the current node's left child is also guaranteed to be rejected. This argument can be easily extended to show that any left-descendant of the current node will be rejected, and it is sufficient to search its right-descendants only. By a similar argument, when X > x ∗ , we find that it is sufficient only to check the current node's left-descendants. Finally, since both Algorithms 3 and 5 simulate Π and search for its first arrival under ϕ , by the construction in Appendix A, the sample returned by both algorithms will follow the desired target distribution.

Analyzing the expected runtime and codelength: Since Algorithm 5 simulates a single branch of T on-sample, its runtime is equal to the depth D of the branch. Furthermore, let N denote the time index of the accepted arrival and H its heap index in T on-sample, which means D = glyph[floorleft] log 2 H glyph[floorright] . It is tempting to use lemmas D.2 and D.3 to in our analysis of N , H and D . However, they do not apply, because they assume no restriction on D and N , while in this section, we condition on the fact that N , H and D identify the arrival in Π that was accepted by Algorithm 5. Hence, we first prove a lemma analogous to lemma D.2 in this restricted case, based on Lemma 4 of Flamich et al. (2022). Note the additional assumptions we need for our result.

Lemma E.1. Let Q and P be distributions over R with unimodal density ratio r = dQ/dP , given to Algorithm 5 as the target and proposal distribution as input, respectively. Assume P has CDF F P . Let H denote the accepted sample's heap index, let D = glyph[floorleft] log 2 H glyph[floorright] , and let B H denote the bounds associated with the accepted node. Then,

Proof. We prove the claim by induction. For H = 1 , D = 0 the claim holds trivially, since B 1 = R , hence P ( B 1 ) = 1 . Assume now that the claim holds for D = glyph[floorleft] log 2 H glyph[floorright] = d -1 for d ≥ 1 , and we prove the statement when D = glyph[floorleft] log 2 H glyph[floorright] = d . Let X 0 , . . . , X d -1 , X d denote the samples simulated along the branch that leads to the accepted node, i.e. X 0 is the sample associated with the root node and X d is the accepted sample. By the definition of the on-sample splitting function and law of iterated expectations, we have

Below, we only examine the inner expectation in the d ≥ 2 case; computing the expectation in the d = 1 case follows mutatis mutandis. First, denote the bounds of the accepted node's parent as B glyph[floorleft] H/ 2 glyph[floorright] = ( α, β ) for some real numbers α < β and define a = F P ( α ) , b = F P ( β ) and U = F P ( X d -1 ) . Since X d -1 | X 1: d -2 ∼ P | B glyph[floorleft] H/ 2 glyph[floorright] , by the generalized probability integral transform we have U ∼ Unif( a, b ) , where Unif( a, b ) denotes the uniform distribution on the interval ( a, b ) . The two possible intervals from which Algorithm 5 will choose are ( α, X d -1 ) and ( X d -1 , β ) , whose measures are P (( α, X d -1 )) = F P ( X d -1 ) -F P ( α ) = U -a and similarly P (( X d -1 , β )) = b -U . Then, in the worst case, the algorithm always picks the bigger of the two intervals, thus P ( B H ) ≤ max { U -A,B -U } , from which we obtain the bound

Plugging this into Equation (195), we get

where the second inequality follows from the inductive hypothesis, which finishes the proof.

Fortunately, lemma D.3 does not depend on this condition, so we can modify it mutatis mutandis so that it still applies in this latter case as well, and we restate it here for completeness:

Lemma E.2. Let N,D,H,B H be defined as above. Then

Proof. Equation (199) follows via the same argument as Equation (181), and Equation (200) follows via the same argument as lemma D.3, combined with lemma E.1.

Expected runtime: Combining Equation (200) with Equation (143) yields

which proves Theorem 3.6. In the channel simulation case, taking a pair of correlated random variables x , y ∼ P x , y and setting Q ← P x | y and P ← P x , and taking expectation over y , we get

Entropy coding the heap index: Let ν ∈ T on-sample now denote the node accepted and returned by Algorithm 5. We will encode ν using its heap index H . As we saw in Appendix D, H can be decomposed into two parts: the depth D and the residual R :

Dealing with the second term first, we note that the maximum entropy distribution for a random variable Z over the positive integers with the moment constraint E [ Z ] = 1 /p is the geometric distribution with rate p . Thus, based on Equation (202), we set

and

Then, we have

from which we get

where Equation (208) follows from the fact that the term -1 -p p log 2 (1 -p ) is bounded from above by log 2 e , which it attains as I [ x ; y ] →∞ .

For the H [ R | D ] term, we begin by noting that there is a particularly natural encoding for R given a particular run of the algorithm. Let B ( D ) 1 , . . . , B ( D ) 2 D be the bounds associated with the depthD nodes of T on-sample | D , the split-on-sample BSP tree given D . Then, since B ( D ) 1 , . . . , B ( D ) 2 D form a P -essential partition of Ω , we can encode R by encoding the bounds associated with the accepted sample, B ( D ) R . Here, we can borrow a technique from arithmetic coding (Rissanen & Langdon, 1979): we encode the largest dyadic interval that is a subset of B ( D ) R , which can be achieved using -log 2 P ( B ( D ) R )+1 bits. Note, that Algorithm 5 only ever realises a single branch of T on-sample, hence, in this section only, let X 1: ∞ denote the infinite sequence of samples along this branch, starting with X 1 ∼ P at the root. Furthermore, from here onwards, let B d and T d denote the bounds and arrival time associated with X d in the sequence, respectively. Then, by the above reasoning, we have

We now develop a bound on P ( B k ) . To this end, note, that by the definition of GPRS,

Hence,

where eq. (215) follows from Markov's inequality, and eq. (216) follows from the fact that the mean measure of ˜ Π is just a restricted version of Π 's mean measure. For convenience, we now write ∆ k = T k -T k -1 . Then, rearranging the above inequality, we get

Substituting this into eq. (212), we get

We deal with the above two sums separately. In the first sum, we apply Jensen's inequality on the concave function x ↦→-x log 2 x in each summand, from which we get

For the second sum, consider the k th summand first:

To continue, we now examine the integrand:

from which

Substituting this back, we get

Therefore, for the second summand, we get

For the first term, by Fubini's theorem and by the correctness of GPRS, we get

For the second term, note that

Hence,

Thus, we finally get that

In the average case, this yields

Thus, we finally get that the joint entropy is bounded above by

which finally proves Theorem 3.7.

F General GPRS with Binary Search

We finally present a generalized version of branch-and-bound GPRS (Algorithm 6) for more general spaces and remove the requirement that r be unimodal.

Decomposing a Poisson process into a mixture of processes: Let Π be a process over R + × Ω as before, with spatial measure P , and let Q be a target measure, ϕ = σ ◦ r and U = { ( t, x ) ∈ R + × Ω | t ≤ ϕ ( x ) } as before. Let ˜ Π be Π = Π ∩ U restricted under ϕ and ( ˜ T 1 , ˜ X 1 ) its first arrival. Let { L, R } form an P -essential partiton of Ω , i.e. L ∩ R = ∅ and P ( L ∪ R ) = 1 , and let Π L = Π ∩ R + × L and Π R = Π ∩ R + × R be the restriction of Π to L and R , respectively. Let ˜ Π L = Π L ∩ U and ˜ Π R = Π R ∩ U be the restrictions of the two processes under ϕ as well. Let ˜ µ L ( t ) and ˜ µ R ( t ) be the mean measures of these processes. Thus, the first arrival times in these processes have survival functions

Note, that by the superposition theorem, Π L ∪ Π R = Π , hence the first arrival ( ˜ T 1 , ˜ X 1 ) occurs in either Π L or Π R . Assume now that we have already searched through the points of Π up to time τ without finding the first arrival. At this point, we can ask: will the first arrival occur in Π L , given that ˜ T 1 ≥ τ ? Using Bayes' rule, we find

More generally, assume that the first arrival of Π occurs in some set A ⊆ Ω , and we know that the first arrival time is larger than τ . Then, what is the probability that the first arrival occurs in some set B ⊆ A ? Similarly to the above, we find

Let ( ˜ T L , ˜ X L ) be the first arrival of Π L . Then, the crucial observation is that

This enables us to search for the first arrival of Π under the graph of ϕ using an extended BSP tree construction. At each node, if we reject, we draw a Bernoulli random variable b with mean equal to the probability that the first arrival occurs within the bounds associated with the right child node. Then, if b = 1 , we continue the search along the right branch. Otherwise, we search along the left branch.

Note, however, that in a restricted process Π A , the spatial measure no longer integrates to 1 . Furthermore, our target Radon-Nikodym derivative is r ( x ) · 1 [ x ∈ A ] /Q ( A ) . This means we need to change

the graph ϕ to some new graph ϕ A to ensure that the spatial distribution of the returned sample is still correct. Therefore, for a set A we define the restricted versions of previous quantities:

Then, via analogous arguments to the ones in Appendix A, we find

Similarly, setting d P [ ˜ X A = x ] dP = 1 [ x ∈ A ] · r ( x ) /Q ( A ) , and setting τ = σ A ( r ( x )) , we get

From this, again using similar arguments to the ones in Appendix A, we find

G Necessary Quantities for Implementing GPRS in Practice

Ultimately, given a target-proposal pair ( Q,P ) with density ratio r , we would want an easy-toevaluate expression for the appropriate stretch function σ or σ -1 to plug directly into our algorithms. Computing σ requires computing the integral in Equation (5) and finding σ -1 requires solving the non-linear ODE in Equation (4), neither of which is usually possible in practice. Hence, we usually resort to computing σ -1 numerically by using an ODE solver for Equation (4).

In any case, we need to compute w P and w Q , which are analytically tractable in the practically relevant cases. Hence, in this section, we give closed-form expressions for w P and w Q for all the examples we consider and give closed-form expressions for σ and σ -1 for some of them. If we do not give a closed-form expression of σ , we use numerical integration to compute σ -1 instead.

Note on notation: In this section only, we denote the natural logarithm as ln and exponentiation with respect to the natural base as exp .

G.1 Uniform-Uniform Case

Let P be the uniform distribution over an arbitrary space Ω and Q a uniform distribution supported on some subset X ⊂ Ω , with P ( X ) = C for some C ≤ 1 Then,

Note that using GPRS in the uniform-uniform case is somewhat overkill, as in this case, it is simply equivalent to standard rejection sampling.

G.2 Triangular-Uniform Case

Let P = Unif(0 , 1) and for some numbers 0 < a < c < b < 1 , let Q be the triangular distribution, defined by the PDF

In this parameterization, the distribution has support on ( a, b ) . Moreover, it is linearly increasing on ( a, c ) and linearly decreasing on ( c, b ) , with the mode located at c . For convenience, let glyph[lscript] = b -a . Then,

Additionally, in this case we find

where ϕ ˜ T and M ˜ T denote the characteristic and the moment-generating functions of ˜ T , respectively. Furthermore, Si , Ci and Ei are the sine, cosine and exponential integrals, defined as

Finally, G 3 , 1 1 , 3 is the Meijer-G function (Sections 5.1-3; Bateman & Erdélyi, 1953) Interestingly, we can see that in this case, while the expectation of ˜ T is finite, we find that V [ ˜ T ] = ∞ . Thus, by Equation (72), we find that V [ N ] = ∞ as well, where N is the number of samples simulated by Algorithm 3.

G.3 Finite Discrete-Discrete / Piecewise Constant Case

Without loss of generality, let Q be a discrete distribution over a finite set with K elements defined by the probability vector ( r 1 < r 2 < . . . < r K ) and P = Unif([1 : K ]) the uniform distribution on K elements. Note that any discrete distribution pair can be rearranged into this setup. Now, we can embed this distribution into [0 , 1] by mapping P to ˆ P = Unif(0 , 1) the uniform distribution over [0 , 1] and mapping Q to ˆ Q with density

Then, we can draw a sample x from Q by simulating a sample ˆ x ∼ ˆ Q and computing x = K · glyph[floorleft] ˆ x/K glyph[floorright] +1 . Hence, we can instead reduce the problem to sampling from a piecewise constant distribution. Thus, let us now instead present the more general case of arbitrary piecewise constant distributions over [0 , 1] , with ˜ Q defined by probabilities ( q 1 < q 2 < . . . < q K ) and corresponding piece widths ( w 1 < w 2 < . . . < w K ) . We require, that ∑ k =1 q k = ∑ K k =1 w k = 1 . Then, the density is

Define r k = q k /w k . Then,

G.4 Diagonal Gaussian-Gaussian Case

Without loss of generality, let Q = N ( µ, σ 2 I ) and P = N (0 , I ) be d -dimensional Gaussian distributions with diagonal covariance. As a slight abuse of notation, let N ( x | µ, σ 2 I ) denote the probability density function of a Gaussian random variable with mean µ and covariance σ 2 I evaluated where we defined

at x . Then, when σ 2 < 1 , we have

Unfortunately, in this case, it is unlikely that we can solve for the stretch function analytically, so in our experiments, we solved for it numerically using Equation (4).

G.5 One-dimensional Laplace-Laplace Case

Without loss of generality, let Q = L ( µ, b ) and P = L (0 , 1) be Laplace distributions. As a slight abuse of notation, let L ( x | µ, b ) denote the probability density function of a Laplace random variable with mean µ and scale b evaluated at x . Then, for b < 1 we have

from which

Similarly to the Gaussian case, we resorted to numerical integration using Equation (4) to solve for σ -1 .

H Experimental Details

Comparing Algorithm 3 versus Global-bound A* coding: We use a setup similar to the one used by Theis & Yosri (2022). Concretely, we assume the following model for correlated random variables x , µ :

From this, we find that the marginal on x must be P x = N (0 , σ 2 +1) . The mutual information is I [ x ; µ ] = 1 2 log 2 ( 1 + σ 2 ) bits, which is what we plot as I [ x ; µ ] in the top two panels in Figure 2.

For the bottom panel in Figure 2, we fixed a standard Gaussian prior P = N (0 , 1) , fixed K = D KL [ Q ‖ P ] and linearly increased R = D ∞ [ Q ‖ P ] . To find a target that achieves the desired values for these given divergences, we set its mean and variances as

where W is the principal branch of the Lambert W -function (Corless et al., 1996).

We can derive this formula by assuming we wish to find µ and σ 2 such that for fixed numbers K and R , and q ( x ) = N ( x | µ, σ 2 ) , p ( x ) = N ( x | 0 , 1) . Then, we have that

We know that

From these, we get that

Setting these equal to each other

where we made the substitutions A = 2log R -2 K -1 and B = 2log R -1 .

I Rejection sampling index entropy lower bound

Assume that we have a pair of correlated random variables x , y ∼ P x , y and Alice and Bob wish to realize a channel simulation protocol using standard rejection sampling as given by, e.g. Algorithm 2. Thus, when Alice receives a source symbol y ∼ P y , she sets Q = P x | y as the target and P = P x as the proposal for the rejection sampler. Let N denote the index of Alice's accepted sample, which is also the number of samples she needs to draw before her algorithm terminates. Since each acceptance decision is an independent Bernoulli trial in standard rejection sampling, N follows a geometric distribution whose mean equals the upper bound M used for the density ratio (Maddison, 2016). The lower bound on the optimal coding cost for N is given by its entropy

where the inequality follows by omitting the first term, which is guaranteed to be positive since x ↦→-x log 2 x is positive on (0 , 1) . Hence, by using the optimal upper bound on the density ratio M ∗ = exp 2 ( D ∞ [ Q ‖ P ]) and plugging it into the formula above, we find that

Now, taking expectation over y , we find

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new relative entropy coding algorithm for compression without quantization. Compression without quantization is an exciting line of research that tries to eliminate training-test mismatches in learned compression by avoiding discrete representations altogether. Consequently, one can losslessly compress data in a continuous latent variable model, such as a VAE, without quantizing the latent representation. The downside of existing algorithms for relative entropy coding is that their runtimes are intolerably slow. The paper makes a big step towards more runtime efficiency by designing new efficient rejection sampling approaches for transmitting samples from a variational posterior, especially in cases where the likelihood ratio between prior and posterior is uni-modal.\n\nSTRENGTHS:\n+ Relative entropy coding is a somewhat under-appreciated but up-and-coming field of research at the intersection between information theory and machine learning. There are only a few papers in this domain. (For example, recent advances made compression in diffusion models possible [Theis et al., 2022].) \n+ The paper significantly improves runtime efficiency for these algorithms, paving the way toward its scalable deployment. Compared to runtime complexities exponential in the Renyi divergence, it achieves a runtime *linear* in the KL divergence between the target and proposal distributions. Notably, the involved algorithm is conceptually much simpler (and in practice faster) than relative entropy coding with A* coding, which achieves the same asymptotic scaling. \n+ Beyond its base version, the paper also proposes two computationally much more efficient extensions: parallel GPRS and branch&bound GPRS. Both approaches rely on non-trivial mathematical insights, e.g., additivity of a Poisson process. \n+ Moreover, in one-dimensional channel simulation problems where the density ratio is unimodal, the algorithm displays the theoretically-optimal runtime (unlike existing approaches such as A* sampling)\n+ The paper shows a nice combination of solid empirical results and theory. For example, most claims on the runtime complexities and theoretical guarantees (finite runtime) are proven rigorously.\n\nWEAKNESSES:\nSee questions below. There are no immediate weaknesses evident, but some clarifying questions need to be addressed.\n\nQUESTIONS:\n* Could the authors discuss in depth the commonalities and differences to a recent paper with a similar title, “adaptive greedy rejection sampling” (https://arxiv.org/pdf/2304.10407.pdf) that the authors already mentioned? \n* The presentation at the beginning of Section 3 is very dense. For example, for an arbitrary P and Q, how is w_p approximated—analytically or based on samples? How does that affect the numerical solvability of Eq. 2?\n* It appears that the stretching function sigma is learned anew for every datum. How does this aspect affect the runtime, and is it respected in the scaling analysis? A discussion would be adequate. \n* Parallelization and branch & bound: it appears that the parallelization affects the cost to *encode* data only, but not to *decode* data. Provided this is correct, could the authors explain why the encoding step is a bigger concern than decoding? In practice (e.g., video streaming), one mostly cares about reducing *decoding* complexity. \n* Could the authors also comment on how they propose to simulate from the Poisson process restricted to subregions? It is unclear why this saves in runtime performance since sampling from a spatially-restricted Poisson process again amounts to rejection sampling. \n\nWriting suggestion:\n* It took me a long time to understand why a temporal poisson process construction was used, where the original communication problem is time-independent. Maybe just say early on that including a temporal poisson process replaces rejection sampling with a deterministic criterion, a trick that was previously proposed for numerical integration (Maddison).\n\nLIMITATIONS:\nLimitations in terms of scaling and speed are sufficiently addressed. Societal impacts are a less important topic for general-purpose data transmission protocols.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of representing a target distribution using the least possible number of bits. The authors refined the idea of encoding a sample from the target distribution as the first sample from the proposal distribution that passes a rejection sampling condition. The refined approach is shown to achieve runtime optimality.\n\nSTRENGTHS:\nThe idea of using a Poisson process to encode a target distribution is powerful. The strategy is well known but further investigations and attempts to make it more feasible could have an impact in various domains.\n\nWEAKNESSES:\nThe authors' technical contribution. could have been outlined more precisely. It is hard to understand why the new strategy is expected to be better than the A* algorithm or naive rejection sampling. The authors should have explained why \"a criterion that does depend on the time variable\" is better.\n\nIn the introduction, the authors say that the distribution is assumed to be unimodal and 1 dimensional. This is a strong assumption and it is unclear why the proposed method requires it. Similar assumptions are made in Flemich 22. The authors may have specified whether these are also required in Maddison 16.\n\nThe authors should have found a way of testing the algorithm on real-world data.\n\nThe text could be more curated and structured.\n\nQUESTIONS:\n- What are the requirements the target and proposal distributions should fulfil? Why do the distributions need to be 1-dimensional? Is this only for theoretical purposes (i.e. to estimate the runtime bounds)? Or would it be also a practical constraint?\n- How expansive is it to solve the ODE for computing \\sigma? Should this be included in the total runtime?\n- How is the density ratio estimated? Is the method feasible if the target distribution is unavailable?\n- What are the key differences between the proposed algorithm, A*, and the approach described in Maddison 16? Why is rejection sampling supposed to be better than importance sampling? It would be good to include an intuitive explanation of why and when GPRS is faster.\n\nLIMITATIONS:\nThe authors do not discuss the limitation of the proposed approach. It is unclear what assumptions are required for the proofs. The theoretical runtime of related methods is not reported.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the channel simulation problem, which finds applications in stochastic lossy compression. In contrast to the importance sampling approach A* coding (Flamich et.al. 2022) for channel simulation with Poisson processes, this work adopts a rejection sampling method. They demonstrate that the standard sampling approach may yield suboptimal code lengths. As an alternative, the authors propose a novel greedy rejection sampling algorithm called the Greedy Rejection Sampling (GPRS) algorithm. This algorithm composites the density ratio $r = \\frac{dQ}{dP}$ with an invertible function referred to as the \"stretch\" function $\\sigma$ that utilizes the temporal structure of the Poisson process without changing the target distribution.  Theoretical findings establish the correctness and optimality of the proposed GPRS algorithm in terms of code length. Furthermore, the expected runtime of GPRS matches the runtime of standard rejection sampling.\n\nIn addition to the vanilla GPRS, the authors introduce parallel and Branch-and-bound variants to further enhance the runtime of the algorithm. Specifically, the Branch-and-bound GPRS demonstrates a provable runtime improvement over A* coding when the density ratio exhibits unimodal characteristics. To validate the theoretical results, the authors conduct several toy experiments.\n\nSTRENGTHS:\n- Mathematical derivations look rigorous. \n- This paper is well-written and easy to follow.\n- Theoretical results seem to be compelling. The authors successfully demonstrate, through rigorous analysis, that the GPRS algorithm outperforms both standard rejection sampling and A* coding in terms of either runtime or code length in some cases.\n\nWEAKNESSES:\n- Some baselines and closely related works are not compared theoretically or empirically (See the first two bullets in Questions).\n\nQUESTIONS:\n- GPRS is thoroughly compared to A* coding both theoretically and experimentally. However, it is unclear why the baseline standard rejection sampling (Algorithm 2) is not empirically compared in Section 4 to validate the theoretical assertion that GPRS enhances the code length efficiency compared to standard RS.\n- Additionally, while the more recent works dithered quantization methods (DQ) and greedy rejection coding (GRC) are briefly mentioned in the related works section, there is a lack of theoretical or empirical comparisons with GPRS. Although these methods may have distinct formulations and proof techniques, conducting some comparisons would be valuable in justifying the significance of GPRS and its potential advantages over alternative approaches.\n- GPRS needs to numerically solve an ODE for $\\sigma^{-1}$ before the time steps. What about the extra cost in runtime?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper investigates the problem of one-shot channel simulation, which can be used as lossy compression without involving quantization.  
A new rejection sampling algorithm called greedy Poisson rejection sampling (GPRS) is proposed. Then,  a parallelized and a branch-and-bound variant is proposed. Those algorithms are analyzed regrading both correctness and runtime.  
Toy experiments on one-dimensional problems show that GPRS compares
favourably against A* coding, the current state-of-the-art channel simulation protocol.

STRENGTHS:
The proposed GPRS is properly positioned against previous and concurrent works, and contains sufficient novelty.    
The proposed method has much better runtime complexity compared with previous SOTA A* coding.  
The manuscript is well written and enough background information is provided for general readers.

WEAKNESSES:
Though this manuscript is a theoretical contribution, it would be better to discuss more about promising application scenarios and current gaps regarding both performance and efficiency, which can encourage more future works in this field and facilitate more applications of similar techniques.

QUESTIONS:
L331-334, please explain in more detail about the application scenario of efficient channel simulation protocol for multivariate Gaussians.

LIMITATIONS:
see above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new algorithm for lossy compression, using ideas from Poisson rejection sampling.\n\nGiven a sample $y \\sim P_y$, Alice wants to communicate the smallest number of bits possible such that Bob can simulate $x \\sim P_{x | y}$, when Alice and Bob have access to the distribution $P_{x, y}$ (and shared randomness to allow simulation).\n\nThe proposed algorithm achieves an almost optimal codelength by rejection sampling -- by introducing a temporal random variable t, Alice can communicate the hitting time $t$ for which $(t, X_t)$ lies under an appropriately defined function $\\phi$. Variations of this algorithm are suggested: one variation is to parallelize using $L$ servers to reduce the runtime by a factor of $L$, and another binary search variant which gives an exponential improvement when the Radon-Nikodym derivative of $dP_{x|y} / dP_{x}$ is bounded.\n\nSTRENGTHS:\n- The paper has several nice ideas from information theory, and seems to be the first that can provably achieve runtimes predicted by Li and El Gamal.\n\n- The algorithm is built on A* coding, but is more robust, in the sense that while the runtime of A* coding depends on the Renyi entropy between $P_{x|y}$ and $P{x}$, the runtime of the proposed algorithm is proportional to the KL divergence between them, which may be much smaller and finite even when the Renyi entropy is infinite.\n\nWEAKNESSES:\n- The rejection sampler requires access to likelihoods of the conditional distribution and marginal, and hence it's not clear how much this can be generalized.\n\n- The inverse function in Eqn (2) is solved numerically. How would this be computed for distributions with harder CDFs? \n\n- All experiments are toy experiments on Gaussian, Binomial variables etc.\n\n- The experiments are for scalar random variables, and extending to higher dimensions would be extremely non-trivial. The authors state that extending this to multivariate Gaussians is an open problem.\n\n- The paper tries to establish connections with areas like generative modelling by arguing that source compression is useful in VAEs/ latent diffusion models, but this is probably not correct. The works of Theis et al for example use generative models for compression, not the other way around.\n\nMinor:\n- The name GPRS for a communication protocol is probably a poor choice given that it's an existing standard.\n- $\\mathbb{V}$ in Theorem 3.1 is undefined.\n\nQUESTIONS:\nQuestions are listed in the weaknesses section.\n\nMy main concern is that this is probably a poor fit for NeurIPS. The theorems are nice, but the impact to a NeurIPS audience is probably extremely limited, given that the algorithms are only feasible for explicit likelihood models and scalar random variables.\n\nLIMITATIONS:\nThere needs to be more discussion about the limitations of this work, especially if the authors are going to mention connections between the proposed algorithm and compression of neural networks. The connections to generative modelling are extremely weak.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}