SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Without loss of generality, let  $Q = \mathcal{L}(\mu, b)$  and  $P = \mathcal{L}(0, 1)$  be Laplace distributions. As a slight abuse of notation, let  $\mathcal{L}(x \mid \mu, b)$  denote the probability density function of a Laplace random variable with mean  $\mu$  and scale  $b$  evaluated at  $x$ . Then, for  $b < 1$  we have
$$
r (x) = \frac {1}{b} \exp \left(- \frac {| x - \mu |}{b} + | x |\right) \tag {294}
$$
$$
w _ {P} (h) = \left\{ \begin{array}{l l} 1 - \exp \left(\frac {b \ln (b h)}{1 - b}\right) \cosh \left(\frac {\mu}{1 - b}\right) & \text {i f} \ln h \leq - \frac {| \mu |}{b} - \ln b \\ \exp \left(\frac {b ^ {2} \ln (b h) - | \mu |}{1 - b ^ {2}}\right) \sinh \left(- \frac {b (\ln (b h) - | \mu |)}{1 - b ^ {2}}\right) & \\ = \frac {1}{2} \left((b h) ^ {\frac {- b}{1 + b}} \exp \left(- \frac {| \mu |}{1 + b}\right) - (b h) ^ {\frac {b}{1 - b}} \exp \left(- \frac {| \mu |}{1 - b}\right)\right) & \text {i f} \ln h > - \frac {| \mu |}{b} - \ln b \end{array} \right. \tag {295}
$$
$$
w _ {Q} (h) = \left\{ \begin{array}{l l} 1 - \exp \left(\frac {\ln (h b)}{1 - b}\right) \cosh \left(\frac {\mu}{1 - b}\right) & \text {i f} \ln h \leq - \frac {| \mu |}{b} - \ln b \\ 1 - \exp \left(\frac {\ln (h b) - | \mu |}{1 - b ^ {2}}\right) \cosh \left(\frac {b (\ln (h b) - | \mu |)}{1 - b ^ {2}}\right) \\ = 1 - \frac {1}{2} (b h) ^ {\frac {1}{1 - b}} \exp \left(- \frac {| \mu |}{1 - b}\right) - \frac {1}{2} (b h) ^ {\frac {1}{1 + b}} \exp \left(- \frac {| \mu |}{1 + b}\right) & \text {i f} \ln h > - \frac {| \mu |}{b} - \ln b, \end{array} \right. \tag {296}
$$
$$
\left(\sigma^ {- 1}\right) ^ {\prime} = \left\{ \begin{array}{l l} 1 - \sigma^ {- 1} + \underbrace {\left(b ^ {\frac {b}{1 - b}} - b ^ {\frac {1}{1 - b}}\right)} _ {\geq 0 \text {o n} [ 0, 1 ]} \cosh \left(\frac {\mu}{1 - b}\right) \left(\sigma^ {- 1}\right) ^ {\frac {1}{1 - b}} & \text {i f} \ln h \leq - \frac {| \mu |}{b} - \ln b \\ 1 + \frac {1}{2} \left(b ^ {\frac {b}{1 - b}} - b ^ {\frac {1}{1 - b}}\right) \exp \left(- \frac {| \mu |}{1 - b}\right) \left(\sigma^ {- 1}\right) ^ {\frac {1}{1 - b}} \\ - \frac {1}{2} \left(b ^ {- \frac {b}{1 + b}} + b ^ {\frac {1}{1 + b}}\right) \exp \left(- \frac {| \mu |}{1 + b}\right) \left(\sigma^ {- 1}\right) ^ {\frac {1}{1 + b}} & \text {o t h e r w i s e} \end{array} \right. \tag {297}
$$
Similarly to the Gaussian case, we resorted to numerical integration using Equation (4) to solve for  $\sigma^{-1}$ .
H Experimental Details
Comparing Algorithm 3 versus Global-bound  $\mathbf{A}^*$  coding: We use a setup similar to the one used by Theis & Yosri (2022). Concretely, we assume the following model for correlated random variables  $\mathbf{x}, \mu$ :
$$
P _ {\mu} = \mathcal {N} (0, 1) \tag {298}
$$
$$
P _ {\mathbf {x} \mid \mu} = \mathcal {N} (\mu , \sigma^ {2}). \tag {299}
$$
From this, we find that the marginal on  $\mathbf{x}$  must be  $P_{\mathbf{x}} = \mathcal{N}(0,\sigma^2 +1)$ . The mutual information is  $I[\mathbf{x};\mu ] = \frac{1}{2}\log_2\left(1 + \sigma^2\right)$  bits, which is what we plot as  $I[\mathbf{x};\mu ]$  in the top two panels in Figure 2.
For the bottom panel in Figure 2, we fixed a standard Gaussian prior  $P = \mathcal{N}(0,1)$ , fixed  $K = D_{\mathrm{KL}}[Q||P]$  and linearly increased  $R = D_{\infty}[Q||P]$ . To find a target that achieves the desired values for these given divergences, we set its mean and variances as
$$
\sigma^ {2} = \exp (W (A \cdot \exp (B)) - B \tag {300}
$$
$$
\mu = 2 K - \sigma^ {2} + \ln \sigma^ {2} + 1 \tag {301}
$$
$$
A = 2 \ln R - 2 K - 1 \tag {302}
$$
$$
B = 2 \ln R - 1, \tag {303}
$$
where  $W$  is the principal branch of the Lambert  $W$ -function (Corless et al., 1996).
We can derive this formula by assuming we wish to find  $\mu$  and  $\sigma^2$  such that for fixed numbers  $K$  and  $R$ , and  $q(x) = \mathcal{N}(x \mid \mu, \sigma^2)$ ,  $p(x) = \mathcal{N}(x \mid 0, 1)$ . Then, we have that
$$
D _ {\mathrm {K L}} [ q \| p ] = K \quad \text {a n d} \quad \sup  _ {x \in \mathbb {R}} \left\{\frac {q (x)}{p (x)} \right\} = R. \tag {304}
$$
$$
K = D _ {\mathrm {K L}} [ q \| p ] = \frac {1}{2} \left[ \mu^ {2} + \sigma^ {2} - \log \sigma^ {2} - 1 \right] \tag {305}
$$
$$
\log R = \log \sup  _ {x \in \mathbb {R}} \left\{\frac {q (x)}{p (x)} \right\} = \frac {\mu^ {2}}{2 (1 - \sigma^ {2})} - \log \sigma .
$$
From these, we get that
$$
\mu^ {2} = 2 K - \sigma^ {2} + \log \sigma^ {2} + 1 \tag {306}
$$
$$
\mu^ {2} = 2 (1 - \sigma^ {2}) (\log R + \log \sigma)
$$
Setting these equal to each other
$$
2 K - \sigma^ {2} + \log \sigma^ {2} + 1 = 2 \log R + \log \sigma^ {2} - 2 \sigma^ {2} \log R - \sigma^ {2} \log \sigma^ {2}
$$
$$
\sigma^ {2} \log \sigma^ {2} - \sigma^ {2} + 2 \sigma^ {2} \log R = 2 \log R - 2 K - 1
$$
$$
\sigma^ {2} \log \sigma^ {2} + \sigma^ {2} (2 \log R - 1) = A
$$
$$
\sigma^ {2} (\log \sigma^ {2} + B) = A
$$
$$
\sigma^ {2} \log \left(\sigma^ {2} e ^ {B}\right) = A \tag {307}
$$
$$
e ^ {B} \sigma^ {2} \log (\sigma^ {2} e ^ {B}) = A e ^ {B}
$$
$$
e ^ {\log (\sigma^ {2} e ^ {B})} \log (\sigma^ {2} e ^ {B}) = A e ^ {B}
$$
$$
\log (\sigma^ {2} e ^ {B}) = W (A e ^ {B})
$$
$$
\sigma^ {2} = e ^ {W (A e ^ {B}) - B},
$$
where we made the substitutions  $A = 2\log R - 2K - 1$  and  $B = 2\log R - 1$ .
I Rejection sampling index entropy lower bound
Assume that we have a pair of correlated random variables  $\mathbf{x},\mathbf{y}\sim P_{\mathbf{x},\mathbf{y}}$  and Alice and Bob wish to realize a channel simulation protocol using standard rejection sampling as given by, e.g. Algorithm 2. Thus, when Alice receives a source symbol  $\mathbf{y}\sim P_{\mathbf{y}}$ , she sets  $Q = P_{\mathbf{x}|\mathbf{y}}$  as the target and  $P = P_{\mathbf{x}}$  as the proposal for the rejection sampler. Let  $N$  denote the index of Alice's accepted sample, which is also the number of samples she needs to draw before her algorithm terminates. Since each acceptance decision is an independent Bernoulli trial in standard rejection sampling,  $N$  follows a geometric distribution whose mean equals the upper bound  $M$  used for the density ratio (Maddison, 2016). The lower bound on the optimal coding cost for  $N$  is given by its entropy
$$
\mathbb {H} [ N ] = - (M - 1) \log_ {2} \left(1 - \frac {1}{M}\right) + \log_ {2} M \geq \log_ {2} M, \tag {308}
$$
where the inequality follows by omitting the first term, which is guaranteed to be positive since  $x \mapsto -x \log_2 x$  is positive on  $(0, 1)$ . Hence, by using the optimal upper bound on the density ratio  $M^* = \exp_2(D_{\infty}[Q \| P])$  and plugging it into the formula above, we find that
$$
D _ {\infty} [ Q \| P ] \leq \mathbb {H} [ N ]. \tag {309}
$$
Now, taking expectation over  $\mathbf{y}$ , we find
$$
\mathbb {E} _ {\mathbf {y} \sim P _ {\mathbf {y}}} \left[ D _ {\infty} \left[ P _ {\mathbf {x} | \mathbf {y}} \| P _ {\mathbf {x}} \right] \right] \leq \mathbb {H} [ N ]. \tag {310}
$$