SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 3.2 (Fractional Moments of the Index). Let  $Q$  and  $P$  be the target and proposal distributions for Algorithm 3, respectively, and  $r = dQ / dP$  their density ratio. Let  $N$  denote the number of samples simulated by the algorithm before it terminates. Then, for  $\alpha \in (0,1)$  we have
$$
\mathbb {E} \left[ N ^ {\alpha} \right] \leq \frac {1}{1 - \alpha} \left(1 + \frac {D _ {\infty} [ Q \| P ]}{\log_ {2} e}\right) \exp_ {2} \left(\alpha D _ {\frac {1}{1 - \alpha}} [ Q \| P ]\right), \tag {7}
$$
where  $D_{\alpha}[Q\| P] = \frac{1}{\alpha - 1}\log_2\mathbb{E}_{Z\sim Q}\left[r(Z)^{\alpha -1}\right]$  is the Renyi  $\alpha$  -divergence of  $Q$  from  $P$
GPRS induces a channel simulation protocol: We use an analogous solution to the standard rejection sampling-based scheme in Section 2.2, to obtain a channel simulation protocol for a pair of correlated random variables  $\mathbf{x},\mathbf{y}\sim P_{\mathbf{x},\mathbf{y}}$ . First, we assume that both the encoder and the decoder have access to the same realization of a  $(1,P_{\mathbf{x}})$ -Poisson process  $\Pi$ , which they will use as their shared randomness. In a single round of communication, the encoder receives  $\mathbf{y}\sim P_{\mathbf{y}}$ , sets  $Q\gets P_{\mathbf{x}|\mathbf{y}}$  as the target and  $P\gets P_{\mathbf{x}}$  as the proposal distribution, and runs Algorithm 3 to find the index  $N$  of the accepted arrival in  $\Pi$ . Finally, following Li & El Gamal (2018), they use a zeta distribution  $\zeta (n\mid s)\propto n^{-s}$  with  $s^{-1} = I[\mathbf{x};\mathbf{y}] + 2\log_2e$  to encode  $N$  using entropy coding. Then, the decoder simulates a  $P_{\mathbf{x}|\mathbf{y}}$ -distributed sample by looking up the spatial coordinate  $X_{N}$  of  $\Pi$ 's  $N$ th arrival. The following theorem shows that this protocol is optimally efficient; see Appendix B.4 for the proof.
Theorem 3.3 (Expected Codelength). Let  $P_{\mathbf{x},\mathbf{y}}$  be a joint distribution over correlated random variables  $\mathbf{x}$  and  $\mathbf{y}$ , and let  $\Pi$  be the  $(1, P_{\mathbf{x}})$ -Poisson process used by Algorithm 3. Then, the above protocol induced by the algorithm is optimally efficient up to a constant, in the sense that
$$
\mathbb {H} [ \mathbf {x} \mid \Pi ] \leq I [ \mathbf {x}; \mathbf {y} ] + \log_ {2} (I [ \mathbf {x}; \mathbf {y} ] + 1) + 6. \tag {8}
$$
Note, that this result shows we can use GPRS as an alternative to the Poisson functional representation to prove of the strong functional representation lemma (Theorem 1; Li & El Gamal, 2018).
GPRS in practice: The real-world applicability of Algorithm 3 is limited by the following three computational obstacles: (1) GPRS requires exact knowledge of the density ratio  $dQ / dP$  to evaluate  $\varphi$ , which immediately excludes a many practical problems where  $dQ / dP$  is known only up to a normalizing constant. (2) To compute  $\sigma$ , we need to evaluate Equation (4) or (5), which requires us to compute  $w_{P}$  and  $w_{Q}$ , which is usually intractable for general  $dQ / dP$ . Fortunately, in some special cases they can be computed analytically: in Appendix G, we provide the expressions for discrete, uniform, triangular, Gaussian, and Laplace distributions. (3) However, even when we can compute  $w_{P}$  and  $w_{Q}$ , analytically evaluating the integral in Equation (5) is usually not possible. Moreover, solving it numerically is unstable as  $\sigma$  is unbounded. Instead, we numerically solve for  $\sigma^{-1}$  using Equation (4) in practice, which fortunately turns out to be stable.
However, we emphasize that GPRS's raison d'etre is to perform channel simulation using the protocol we outline above or the ones we present in Section 3.1, not general-purpose random variate simulation. Practically relevant channel simulation problems, such as encoding a sample from the latent posterior of a VAE (Flamich et al., 2020) or encoding variational implicit neural representations (Guo et al., 2023; He et al., 2023) usually involve simple distributions such as Gaussians. In these cases, we have readily available formulae for  $w_{P}$  and  $w_{Q}$  and can cheaply evaluate  $\sigma^{-1}$  via numerical integration.
GPRS as greedy search: The defining feature of GPRS is that its acceptance criterion at each step is local, since if at step  $n$  the arrival  $(T_{n},X_{n})$  in  $\Pi$  falls under  $\varphi$ , the algorithm accepts it. Thus it is greedy search procedure. This is in contrast with  $\mathrm{A}^*$  sampling (Maddison et al., 2014), whose acceptance criterion is global, as the acceptance of the  $n$ th arrival  $(T_{n},X_{n})$  depends on all other points of  $\Pi$  in the general case. In other words, using the language of Liu & Verdu (2018), GPRS is a causal sampler, as its acceptance criterion at each step only depends on the samples the algorithm has examined in previous steps. On the other hand,  $\mathrm{A}^*$  sampling is non-causal as its acceptance criterion in each step depends on future samples as well. Surprisingly, this difference between the search criteria does not make a difference in the average runtimes and codelengths in the general case. However, as we show in the next section, GPRS can be much faster in special cases.
3.1 Speeding up the greedy search
This section discusses two ways to improve the runtime of GPRS. First, we show how we can utilize available parallel computing power to speed up Algorithm 3. Second, we propose an advanced search
Algorithm 4: Parallel GPRS with  $J$  available threads.  
Input: Proposal distribution  $P$ , Density ratio  $r = dQ / dP$ , Stretch function  $\sigma$ , Number of parallel threads  $J$ .  
Output: Sample  $X \sim Q$  and its code  $(j^*, N_{j*})$ .  
 $T^*, X^*, j^*, N_{j*} \gets \infty, \text{nil}, \text{nil}, \text{nil}$   
in parallel for  $j = 1, \ldots, J$  do  
 $\Pi_j \gets \text{SimulatePP}(1 / J, P)$   
for  $n_j = 1, 2, \ldots$  do  
 $\left(T_{nj}^{(j)}, X_{nj}^{(j)}\right) \gets \text{next}(\Pi_j)$   
if  $T^* < T_{nj}^{(j)}$  then  
| terminate thread  $j$ .  
end  
if  $T_{nj}^{(j)} < \sigma(r(X_{nj}^{(j)}))$  then  
|  $T^*, X^*, j^*, N_{j*} \gets T_{nj}^{(j)}, X_{nj}^{(j)}, j, n_j$   
| terminate thread  $j$ .  
end  
end  
return  $X^*, (j^*, N_{j*})$
Algorithm 5: Branch-and-bound GPRS on  $\mathbb{R}$  with unimodal  $r$    
Input :Proposal distribution  $P$  Density ratio  $r = dQ / dP$  Stretch function  $\sigma$  Location  $x^{*}$  of the mode of  $r$    
Output:Sample  $X\sim Q$  and its heap index  $H$ $T_0,H,B\gets (0,1,\mathbb{R})$    
for  $d = 1,2,\ldots$  do  $X_{d}\sim P|_{B}$ $\Delta_d\sim \mathrm{Exp}(P(B))$ $T_{d}\leftarrow T_{d - 1} + \Delta_{d}$  if  $T_{d} <   \sigma (r(X_{d}))$  then return  $X_{d},H$    
end   
if  $X_{d}\geq x^{*}$  then  $B\gets B\cap (-\infty ,X_d)$ $H\gets 2H$    
else  $B\gets B\cap (X_d,\infty)$ $H\gets 2H + 1$    
end
strategy when the spatial domain  $\Omega$  has more structure and obtain a super-exponential improvement in the runtime from  $\exp_2(D_{\infty}[Q\| P])$  to  $\mathcal{O}(D_{\mathrm{KL}}[Q\| P])$  in certain cases.
Parallel GPRS: The basis for parallelizing GPRS is the superposition theorem (Kingman, 1992): For a positive integer  $J$ , let  $\Pi_1, \ldots, \Pi_J$  all be  $(1 / J, P)$ -Poisson processes; then,  $\Pi = \bigcup_{j=1}^{J} \Pi_j$  is a  $(1, P)$ -Poisson process. As shown in Algorithm 4, this result makes parallelizing GPRS very simple given  $J$  parallel threads: First, we independently look for the first arrivals of  $\Pi_1, \ldots, \Pi_J$  under the graph of  $\varphi$ , yielding  $\left(T_{N_1}^{(1)}, X_{N_1}^{(1)}\right), \ldots, \left(T_{N_J}^{(J)}, X_{N_J}^{(J)}\right)$ , respectively, where the  $N_j$  corresponds to the index of  $\Pi_j$ 's first arrival under  $\varphi$ . Then, we select the candidate with the earliest arrival time, i.e.  $j^* = \arg \min_{j \in \{1, \ldots, J\}} T_{N_j}^{(j)}$ . Now, by the superposition theorem,  $\left(T_{N_{j^*}}^{(j^*)}, X_{N_{j^*}}^{(j^*)}\right)$  is the first arrival of  $\Pi$  under  $\varphi$ , and hence  $X_{N_{j^*}}^{(j^*)} \sim Q$ . Finally, Alice encodes the sample via the tuple  $(j^*, N_{j^*})$ , i.e. which of the  $J$  processes the first arrival occurred in, and the index of the arrival in  $\Pi_{j^*}$ . See the middle graphic in Figure 1 for an example with  $J = 2$  parallel threads.
Our next results show that parallelizing GPRS results in a linear reduction in both the expectation and variance of its runtime and a more favourable codelength guarantee for channel simulation.
Theorem 3.4 (Expected runtime of parallelized GPRS). Let  $Q, P$  and  $r$  be defined as above, and let  $\nu_{j}$  denote the random variable corresponding to the number of samples simulated by thread  $j$  in Algorithm 4 using  $J$  threads. Then, for all  $j$ ,
$$
\mathbb {E} [ \nu_ {j} ] = \left(\exp_ {2} \left(D _ {\infty} [ Q \| P ]\right) - 1\right) / J + 1 \quad a n d \quad \mathbb {V} [ \nu_ {j} ] \geq \left(\exp_ {2} \left(D _ {\infty} [ Q \| P ]\right) - 1\right) / J + 1. \tag {9}
$$
Theorem 3.5 (Expected codelength of parallelized GPRS). Let  $P_{\mathbf{x},\mathbf{y}}$  be a joint distribution over correlated random variables  $\mathbf{x}$  and  $\mathbf{y}$ , and let  $\Pi_1, \ldots, \Pi_J$  be  $(1 / J, P_{\mathbf{x}})$ -Poisson processes. Then, assuming  $\log_2 J \leq I[\mathbf{x}; \mathbf{y}]$ , parallelized GPRS induces a channel simulation protocol such that
$$
\mathbb {H} [ \mathbf {x} \mid \Pi_ {1}, \dots , \Pi_ {J} ] \leq I [ \mathbf {x}; \mathbf {y} ] + \log_ {2} (I [ \mathbf {x}; \mathbf {y} ] - \log_ {2} J + 1) + 8. \tag {10}
$$
See Appendix C for the proofs. Note that we can use the same parallelisation argument with the appropriate modifications to speed up  $\mathrm{A}^*$  sampling / coding too.
Branch-and-bound GPRS on  $\mathbb{R}$ : We briefly restrict our attention to problems on  $\mathbb{R}$  when the density ratio  $r$  is unimodal, as we can exploit this additional structure to more efficiently search for  $\Pi$ 's first
arrival under  $\varphi$ . Consider the example in the right graphic in Figure 1: we simulate the first arrival  $(T_{1},X_{1})$ , and reject it, since  $T_{1} > \varphi(X_{1})$ . Since  $r$  is unimodal by assumption and  $\sigma$  is increasing,  $\varphi$  is also unimodal. Hence, for  $x < X_{1}$  we must have  $\varphi(x) < \varphi(X_{1})$ , while the arrival time of any of the later arrivals will be larger than  $T_{1}$ . Therefore, none of the arrivals to the left of  $X_{1}$  will fall under  $\varphi$  either! Hence, it is sufficient to simulate  $\Pi$  to the right of  $X_{1}$ , which we can easily do using generalized inverse transform sampling. We repeat this argument for all further arrivals to obtain an efficient search procedure for finding  $\Pi$ 's first arrival under  $\varphi$ , described in Algorithm 5: We simulate the next arrival  $(T_{B},X_{B})$  of  $\Pi$  within some bounds  $B$ , starting with  $B = \Omega$ . If  $T_{B} \leq \varphi(X_{B})$  we accept; otherwise, we truncate the bound to  $B \gets B \cap (-\infty, X_{B})$ , or  $B \gets B \cap (X_{B}, \infty)$  based on where  $X_{B}$  falls relative to  $\varphi$ 's mode. We repeat these two steps until we find the first arrival.
Since Algorithm 5 does not simulate every point of  $\Pi$ , we cannot use the index  $N$  of the accepted arrival to obtain a channel simulation protocol as before. Instead, we encode the search path, i.e. whether we chose the left or the right side of our current sample at each step. Similarly to  $\mathbf{A}^*$  coding, we encode the path using its heap index (Flamich et al., 2022): the root has index  $H_{root} = 1$ , and for a node with index  $H$ , its left child is assigned index  $2H$  and its right child  $2H + 1$ . As the following theorems show, this version of GPRS is, in fact, optimally efficient; see Appendix E for proofs.
Theorem 3.6 (Expected Runtime of GPRS with binary search). Let  $Q$ ,  $P$  and  $r$  be defined as above, and let  $D$  denote the number of samples simulated by Algorithm 5 and  $\lambda = 1 / \log_2(4/3)$ . Then,
$$
\mathbb {E} [ D ] \leq \lambda \cdot D _ {\mathrm {K L}} [ Q \| P ] + \mathcal {O} (1). \tag {11}
$$
Theorem 3.7 (Expected Codelength of GPRS with binary search). Let  $P_{\mathbf{x},\mathbf{y}}$  be a joint distribution over correlated random variables  $\mathbf{x}$  and  $\mathbf{y}$ , and let  $\Pi$  be a  $(1, P_{\mathbf{x}})$ -Poisson process. Then, GPRS with binary search induces a channel simulation protocol such that
$$
\mathbb {H} [ \mathbf {x} \mid \Pi ] \leq I [ \mathbf {x}; \mathbf {y} ] + 2 \log_ {2} (I [ \mathbf {x}; \mathbf {y} ] + 1) + 1 1. \tag {12}
$$
Branch-and-bound GPRS with splitting functions: With some additional machinery, Algorithm 5 can be extended to more general settings, such as  $\mathbb{R}^D$  and cases where  $r$  is not unimodal, by introducing the notion of a splitting function. For a region of space,  $B \subseteq \Omega$ , a splitting function split simply returns a binary partition of the set, i.e.  $\{L, R\} = \text{split}(B)$ , such that  $L \cap R = \emptyset$  and  $L \cup R = B$ . In this case, we can perform a similar tree search to Algorithm 5, captured in Algorithm 6. Recall that  $(\tilde{T}, \tilde{X})$  is the first arrival of  $\Pi$  under  $\varphi$ . Starting with the whole space  $B = \Omega$ , we simulate the next arrival  $(T, X)$  of  $\Pi$  in  $B$  at each step. If we reject it, we partition  $B$  into two parts,  $L$  and  $R$ , using split. Then, with probability  $\mathbb{P}[\tilde{X} \in R \mid \tilde{X} \in B, \tilde{T} \geq T]$ , we continue searching through the arrivals of  $\Pi$  only in  $R$ , and only in  $L$  otherwise. We show the correctness of this procedure in Appendix F and describe how to compute  $\mathbb{P}[\tilde{X} \in R \mid \tilde{X} \in B, \tilde{T} \geq T]$  and  $\sigma$  in practice. This splitting procedure is analogous to the general version of A* sampling/coding (Maddison et al., 2014; Flamich et al., 2022), which is parameterized by the same splitting functions. Note that Algorithm 5 is a special case of Algorithm 6, where  $\Omega = \mathbb{R}$ ,  $r$  is unimodal, and at each step for an interval bound  $B = (a, b)$  and sample  $X \in (a, b)$  we split  $B$  into  $\{(a, X), (X, b)\}$ .
We compare the average and one-shot case efficiency of our proposed variants of GPRS and a couple of other channel simulation protocols in Figure 2. See the figure's caption and Appendix H for details of our experimental setup. The top two plots in Figure 2 demonstrate that our methods' expected runtimes and codelengths align well with our theorems' predictions and compare favourably to other methods. Furthermore, we find that the mean performance is a robust statistic for the binary search-based variants of GPRS in that it lines up closely with the median, and the interquartile range is quite narrow. However, we also see that Theorem 3.7 is slightly loose empirically. We conjecture that the coefficient of the  $\log_2(I[\mathbf{x};\mathbf{y}] + 1)$  term in Equation (12) could be reduced to 1, and that the constant term could be improved as well.
On the other hand, the bottom plot in Figure 2 demonstrates the most salient property of the binary search variant of GPRS: unlike all previous methods, its runtime scales with  $D_{\mathrm{KL}}[Q \| P]$  and not  $D_{\infty}[Q \| P]$ . Thus, we can apply it to a larger family of channel simulation problems, where  $D_{\mathrm{KL}}[Q \| P]$  is finite but  $D_{\infty}[Q \| P]$  is large or even infinite, and other methods would not terminate.
Suppose we have a real-valued random variable  $X$  with CDF  $F$  and inverse CDF  $F^{-1}$ . Then, we can simulate  $X$  truncated to  $[a,b]$  by simulating  $U \sim \mathrm{Unif}[F(a),F(b)]$  and computing  $X = F^{-1}(U)$ .
Algorithm 6: Branch-and-bound GPRS with splitting function.
Input :Proposal distribution  $P$  Density ratio  $r = dQ / dP$  Stretch function  $\sigma$  Splitting function split.
Output: Sample  $X \sim Q$  and its heap index  $H$ $T_0, H, B \gets (0,1,\mathbb{R})$
Figure 2: Left: Binary search GPRS with arbitrary splitting function. Right: Performance comparison of different channel simulation protocols. In each plot, dashed lines indicate the mean, solid lines the median and the shaded areas the 25 - 75 percentile region of the relevant performance metric. We computed the statistics over 1000 runs for each setting. Top right: Runtime comparison on a 1D Gaussian channel simulation problem  $P_{\mathbf{x},\mu}$ , plotted against increasing mutual information  $I[\mathbf{x};\mu]$ . Alice receives  $\mu \sim \mathcal{N}(0,\sigma^2)$  and encodes a sample  $\mathbf{x} \mid \mu \sim \mathcal{N}(\mu, 1)$  to Bob. The abbreviations in the legend are: GPRS – Algorithm 3; PFR – Poisson functional representation / Global-bound A* coding (Li & El Gamal, 2018; Flamich et al., 2022), AS* – Split-on-sample A* coding (Flamich et al., 2022); sGPRS – split-on-sample GPRS (Algorithm 5); dGPRS – GPRS with dyadic split (Algorithm 6). Middle right: Average codelength comparison of our proposed algorithms on the same channel simulation problem as above. UB in the legend corresponds to an upper bound of  $I[\mathbf{x};\mu] + \log_2(I[\mathbf{x};\mu] + 1) + 2$  bits. We estimate the algorithms' expected codelengths by encoding the indices returned by GPRS using a Zeta distribution  $\zeta(n \mid \lambda) \propto n^{-\lambda}$  with  $\lambda = 1 + 1 / I[\mathbf{x};\mathbf{y}]$  in each case, which is the optimal maximum entropy distribution for this problem setting (Li & El Gamal, 2018). Bottom right: One-shot runtime comparison of sGPRS with AS* coding. Alice encodes samples from a target  $Q = \mathcal{N}(m,s^2)$  using  $P = \mathcal{N}(0,1)$  as the proposal. We computed  $m$  and  $s^2$  such that  $D_{\mathrm{KL}}[Q||P] = 2$  bits for each problem, but  $D_{\infty}[Q||P]$  increases. GPRS' runtime stays fixed as it scales with  $D_{\mathrm{KL}}[Q||P]$ , while the runtime of A* keeps increasing.
Poisson processes for channel simulation: Poisson processes were introduced to the channel simulation literature by Li & El Gamal (2018) via their construction of the Poisson functional representation (PFR) in their proof of the strong functional representation lemma. Flamich et al. (2022) observed that the PFR construction is equivalent to a certain variant of  $\mathrm{A}^*$  sampling (Maddison et al., 2014; Maddison, 2016). Thus, they proposed an optimized version of the PFR called  $\mathrm{A}^*$  coding, which achieves  $\mathcal{O}(D_{\infty}[Q||P])$  runtime for one-dimensional unimodal distributions. GPRS was mainly inspired by  $\mathrm{A}^*$  coding, and they are dual constructions to each other in the following sense: Depth-limited  $A^*$  coding can be thought of as an importance sampler, i.e. a Monte Carlo algorithm that returns an approximate sample in fixed time. On the other hand, GPRS is a rejection sampler, i.e. a Las Vegas algorithm that returns an exact sample in random time.
Channel simulation with dithered quantization: Dithered quantization (DQ; Ziv, 1985) is an alternative to rejection and importance sampling-based approaches to channel simulation. DQ exploits that for any  $c \in \mathbb{R}$  and  $U, U' \sim \mathrm{Unif}(-1/2, 1/2)$ , the quantities  $\lfloor c - U \rfloor + U$  and  $c + U'$  are equal in distribution. While DQ has been around for decades as a tool to model and analyze quantization error, Agustsson & Theis (2020) reinterpreted it as a channel simulation protocol and used it to develop a VAE-based neural image compression algorithm. Unfortunately, basic DQ only allows uniform target distributions, limiting its applicability. As a partial remedy, Theis & Yosri (2022) showed DQ could be combined with other channel simulation protocols to speed them up and thus called their approach hybrid coding (HQ). Originally, HQ required that the target distribution be compactly supported, which was lifted by Flamich & Theis (2023), who developed an adaptive rejection sampler using HQ. In a different vein, Hegazy & Li (2022) generalize and analyze a method proposed in the appendix of Agustsson & Theis (2020) and show that DQ can be used to realize channel simulation protocols for one-dimensional symmetric, unimodal distributions.
Greedy Rejection Coding: Concurrently to our work, Flamich et al. (2023) introduce greedy rejection coding (GRC) which generalizes Harsha et al.'s rejection sampling algorithm to arbitrary probability spaces and arbitrary splitting functions, extending the work of Flamich & Theis (2023). Furthermore, they also utilize a space-partitioning procedure to speed up the convergence of their sampler and prove that a variant of their sampler also achieves optimal runtime for one-dimensional problems where  $dQ / dP$  is unimodal. However, the construction of their method differs significantly from ours. GRC is a direct generalization of Harsha et al. (2007)'s algorithm and, thus, a more "conventional" rejection sampler, while we base our construction on Poisson processes. Thus, our proof techniques are also significantly different. It is an interesting research question whether GRC could be formulated using Poisson processes, akin to standard rejection sampling in Algorithm 2, as this could be used to connect the two algorithms and improve both.
Using the theory of Poisson processes, we constructed greedy Poisson rejection sampling. We proved the correctness of the algorithm and analyzed its runtime, and showed that it could be used to obtain a channel simulation protocol. We then developed several variations on it, analyzed their runtimes, and showed that they could all be used to obtain channel simulation protocols. As the most significant result of the paper, we showed that using the binary search variant of GPRS we can achieve  $\mathcal{O}(D_{\mathrm{KL}}[Q\| P])$  runtime for arbitrary one-dimensional, unimodal density ratios, significantly improving upon the previous best  $\mathcal{O}(D_{\infty}[Q\| P])$  bound by A* coding.
There are several interesting directions for future work. From a practical perspective, the most pressing question is whether efficient channel simulation algorithms exist for multivariate problems; finding an efficient channel simulation protocol for multivariate Gaussians would already have far-reaching practical consequences. We also highlight an interesting theoretical question. We found that the most general version of GPRS needs to simulate exactly  $\exp_2(D_{\infty}[Q||P])$  samples in expectation, which matches A* sampling. From Agustsson & Theis (2020), we know that for general problems, the average number of simulated samples has to be at least on the order of  $\exp_2(D_{\mathrm{KL}}[Q||P])$ . Hence we ask whether this lower bound can be tightened to match the expected runtime of GPRS and A* sampling or whether there exists an algorithm that achieves a lower runtime.
We are grateful to Lennie Wells for the many helpful discussions throughout the project, particularly for suggesting the nice argument for Lemma D.2; Peter Wildemann for his advice on solving some of the differential equations involved in the derivation of GPRS and Stratis Markou for his help that aided our understanding of some high-level concepts so that we could derive Algorithm 6. We would also like to thank Lucas Theis for the many enjoyable early discussions about the project and for providing helpful feedback on the manuscript; and Cheuk Ting Li for bringing Liu & Verdu's work to our attention. Finally, we would like to thank Daniel Goc for proofreading the manuscript and Lorenzo Bonito for helping to design Figures 1 and 2 and providing feedback on the early version of the manuscript. GF acknowledges funding from DeepMind.