SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\bar {\alpha} _ {k} (p, q) = \min  _ {\Omega_ {0} \subset \Omega} \left\{\sum_ {y \in \Omega_ {0}} \min  \left\{q (y), 1 - (1 - p (y)) ^ {k} \right\} + \sum_ {x ^ {k} \in \Omega^ {k}} \min  \left\{\prod_ {i = 1} ^ {k} p \left(x _ {i}\right), \sum_ {y \in x ^ {k} \cap \Omega_ {0} ^ {c}} q (y) \right\} \right\}. \tag {9}
$$
In Appendix B, we plot  $\alpha_{k}$  as a function of  $k$  for a few simple pairs of  $(p,q)$ 's as illustrative examples. We note that the upper bound in Lemma 3 is tight for examples considered in Appendix B.
We first prove monotonicity. By definition,
$$
\begin{array}{l} \alpha_ {k} (p, q) = 1 - \min  _ {\pi \in \Pi \left(p ^ {\otimes k}, q\right)} \Pr_ {X ^ {k}, Y \sim \pi} \left(Y \notin S \left(X ^ {k}\right)\right) \\ = \max  _ {\pi \in \Pi \left(p ^ {\otimes k}, q\right)} \Pr_ {X ^ {k}, Y \sim \pi} \left(Y \in S \left(X ^ {k}\right)\right) \\ \end{array}
$$
Moreover, for any  $\pi \in \Pi(p^{\otimes k}, q)$ , we can construct  $\pi' \in \Pi(p^{\otimes k+1}, q)$  by setting
$$
\forall x ^ {k + 1} \in \Omega^ {k + 1}, y \in \Omega , \pi^ {\prime} (x ^ {k + 1}, y) = \pi (x ^ {k}, y) p (x (k + 1)),
$$
i.e., adding and independent sample from  $p$  to  $X^k$ .
$$
\begin{array}{l} \alpha_ {k} (p, q) = \max  _ {\pi \in \Pi \left(p ^ {\otimes k}, q\right)} \Pr_ {X ^ {k}, Y \sim \pi} \left(Y \in S \left(X ^ {k}\right)\right) \\ = \max  _ {\pi \in \Pi (p ^ {\otimes k}, q)} \Pr_ {X ^ {k + 1}, Y \sim \pi^ {\prime}} \left(Y \in S \left(X ^ {k}\right)\right) \\ \leq \max  _ {\pi \in \Pi (p ^ {\otimes k}, q)} \Pr_ {X ^ {k + 1}, Y \sim \pi^ {\prime}} \left(Y \in S (X ^ {k + 1})\right) \\ \leq \max  _ {\pi^ {\prime} \in \Pi (p ^ {\otimes k + 1}, q)} \Pr_ {X ^ {k + 1}, Y \sim \pi^ {\prime}} \left(Y \in S \left(X ^ {k + 1}\right)\right) \\ = \alpha_ {k + 1} (p, q). \\ \end{array}
$$
Next we prove consistency. We start with the case when  $\forall x \in \Omega, q(x) / p(x) < \infty$ . To prove this, we will show that Algorithm 2 with  $\rho_{\max} = \max_{x \in \Omega} q(x) / p(x)$  satisfies
$$
\lim_{k\to \infty}\alpha (\pi_{\rho_{\max}}^{\mathrm{K - SEQ}}) = 1.
$$
Since  $\alpha (\pi_{\rho_{\max}}^{\mathrm{K - SEQ}})\leq \alpha_k(p,q)$ , the above equation implies  $\lim_{k\to \infty}\alpha_k(p,q) = 1$ . Notice that by Lemma 4 and Theorem 1,  $\pi_{\rho_{\max}}^{\mathrm{K - SEQ}}$  is a valid coupling, and
$$
\alpha \left(\pi_ {\rho_ {\max }} ^ {\mathrm {K} - \mathrm {S E Q}}\right) = 1 - \left(1 - \beta_ {p, q} \left(\rho_ {\max }\right)\right) ^ {k},
$$
where  $\beta_{p,q}(\rho) = \sum_{x\in \Omega}\min (p(x),\frac{q(x)}{\rho})\geq 1 / \rho_{\max} > 0$ . Taking  $k\to \infty$  concludes the proof.
For the case when  $q(x) / p(x)$  is unbounded, there exists  $x \in \Omega$  such that  $q(x) > 0$  and  $p(x) = 0$ . Let
$$
p _ {\text {o f f}} = \sum_ {x \in \Omega} \mathbb {1} \left\{p (x) = 0 \right\} q (x).
$$
Let  $x_0$  be such that  $p(x_0) > 0$ . We define  $q'$  such that
$$
q ^ {\prime} = \left\{ \begin{array}{l l} 0, & \text {i f} p (x) = 0, \\ q (x), & \text {i f} p (x) > 0 \text {a n d} x \neq x _ {0}, \\ q (x) + p _ {\text {o f f}} & \text {i f} x = x _ {0}. \end{array} \right.
$$
Then we have  $d_{\mathrm{TV}}(q,q^{\prime}) = p_{\mathrm{off}}$ , and hence by subadditivity of transport cost,
$$
\alpha_ {k} (p, q) \geq \alpha_ {k} (p, q ^ {\prime}) - p _ {\mathrm {o f f}}.
$$
Moreover, we have  $\forall x\in \Omega ,q'(x) / p(x) <   \infty$  . Hence
$$
\lim  _ {k \rightarrow \infty} \alpha_ {k} (p, q) \geq \lim  _ {k \rightarrow \infty} \alpha_ {k} (p, q ^ {\prime}) - p _ {\text {o f f}} = 1 - p _ {\text {o f f}} = \sum_ {x \in \Omega} \mathbb {1} \left\{p (x) > 0 \right\} q (x).
$$
For the upper bound, it would be enough to show that for any  $\pi \in \Pi(p^{\otimes k}, q)$ , and any  $\Omega_0 \subset \Omega$ , we have
$$
\begin{array}{l} \operatorname * {P r} \left(Y \in S (X ^ {k})\right) \leq \sum_ {y \in \Omega_ {0}} \min  \left\{q (y), 1 - (1 - p (y)) ^ {k} \right\} + \sum_ {x ^ {k} \in \Omega^ {k}} \min  \{\prod_ {i = 1} ^ {k} p (x _ {i}), \sum_ {y \in S (x ^ {k}) \cap \Omega_ {0} ^ {c}} q (y) \}. \\ \Pr \left(Y \in S \left(X ^ {k}\right)\right) \\ = \sum_ {y \in \Omega} \sum_ {x ^ {k} \in \Omega^ {k}} \Pr \left(X ^ {k} = x ^ {k}, Y = y\right) \cdot \mathbb {1} \left\{y \in S (x ^ {k}) \right\} \\ = \sum_ {y \in \Omega_ {0}} \sum_ {x ^ {k} \in \Omega^ {k}} \Pr \left(X ^ {k} = x ^ {k}, Y = y\right) \cdot \mathbb {1} \left\{y \in S \left(x ^ {k}\right) \right\} \\ + \sum_ {y \in \Omega_ {0} ^ {c}} \sum_ {x ^ {k} \in \Omega^ {k}} \Pr \left(X ^ {k} = x ^ {k}, Y = y\right) \cdot \mathbb {1} \left\{y \in S \left(x ^ {k}\right) \right\} \\ = \sum_ {y \in \Omega_ {0}} \Pr \left(y \in S \left(x ^ {k}\right), Y = y\right) + \sum_ {x ^ {k} \in \Omega^ {k}} \sum_ {y \in S \left(x ^ {k}\right) \cap \Omega_ {0} ^ {c}} \Pr \left(X ^ {k} = x ^ {k}, Y = y\right) \\ \leq \sum_ {y \in \Omega_ {0}} \min  \left\{\Pr \left(y \in S \left(x ^ {k}\right)\right), q (y) \right\} + \sum_ {x ^ {k} \in \Omega^ {k}} \min  \left\{\Pr \left(X ^ {k} = x ^ {k}\right), \sum_ {y \in S \left(x ^ {k}\right) \cap \Omega_ {0} ^ {c}} q (y) \right\} \\ = \sum_ {y \in \Omega_ {0}} \min  \left\{1 - (1 - p (y)) ^ {k}, q (y) \right\} + \sum_ {x ^ {k} \in \Omega^ {k}} \min  \left\{\prod_ {i = 1} ^ {k} p \left(x _ {i}\right), \sum_ {y \in S \left(x ^ {k}\right) \cap \Omega_ {0} ^ {c}} q (y) \right\}. \\ \end{array}
$$
For the lower bound, we show that K-SEQ achieves an acceptance probability of at least  $(1 - (1 - 1 / k)^k) \bar{\alpha}_k(p,q)$ , see Eq. (11), implying the lower bound guarantee.
B Comparison between  $\alpha (\pi_{\rho^*}^{\mathrm{K - SEQ}})$  and  $\alpha_{k}$  for simple examples.
We illustrate the acceptance probabilities for our proposed token-level selection algorithms using a few simple examples and plot them in Figures 4 and 5. The analysis for these simple distributions is presented in Appendix B.1 and Appendix B.2.
Pairs of Bernoulli distributions. Let  $\operatorname{Ber}(b)$  be a Bernoulli distribution with probability  $b$  of getting a head. In Figure 4, we plot the acceptance probability comparison between OTM- $k$  and K-SEQ for different Bernoulli distributions  $q = \operatorname{Ber}(b)$  as a function of  $k$  when  $p = \operatorname{Ber}(0.25)$ . Note that when  $p = q$  ( $b = 0.25$ ), the acceptance probability is always one for both methods. When  $p \neq q$ , the acceptance probabilities for both methods increase as  $k$  increases before they reach one. When  $b = 0.1$  or 0.75, K-SEQ has a worse acceptance probability compared to the OTM- $k$  algorithm. When  $b = 1$ , the two algorithms have the same performance.
Figure 4: Acceptance probability comparison between OTM-  $k$  and K-SEQ when  $p = \mathrm{Ber}(0.25)$  and  $q = \mathrm{Ber}(b)$ .
Figure 5: Optimal acceptance probability  $(\alpha_{k})$  as a function of  $k$  when  $p = U(d)$  for  $d = 120$  and  $q = U(d_{q})$ .
Pairs of uniform distributions. Let  $U(d)$  denote a uniform distribution over  $[d]$ . In Figure 5, we plot the optimal acceptance probability for different uniform functions  $q$  as a function of  $k$ . For these distributions, it can be shown that K-SEQ achieves the optimal acceptance probability  $\alpha_{k}$ . Hence only  $\alpha_{k}$  is plotted. Observe that all acceptance probabilities are monotonically increasing and tend to one when  $k \to \infty$ , as stated in Lemma 1.
B.1 Calculations for  $\alpha_{k}$
In this section, we provide a sketch of optimal acceptance probability calculations for results in Figures 4 and 5.
Figure 4:  $\mathbf{Ber}(p)$  and  $\mathbf{Ber}(q)$ . The optimal acceptance probability is
$$
\alpha_ {k} (\operatorname {B e r} (p), \operatorname {B e r} (q)) = \min  (q, 1 - (1 - p) ^ {k}) + \min  (1 - q, 1 - p ^ {k}). \tag {10}
$$
Setting  $\Omega_0 = \{0,1\}$  in Lemma 3 yields the upper bound. For the lower bound observe that since  $\Omega = \{0,1\}$ ,  $\mathbb{1}\left\{y\notin S(x^k)\right\} < 1$  if and only if  $x^{k}$  is  $0^{k}$  or  $1^{k}$ . Hence,
$$
\begin{array}{l} \alpha_ {k} (\operatorname {B e r} (p), \operatorname {B e r} (q)) = \pi \left(X ^ {k} \notin \{0 ^ {k}, 1 ^ {k} \}\right) + \max  _ {\pi} \left\{\pi \left(Y = 0, X ^ {k} = 0 ^ {k}\right) + \pi \left(Y = 1, X ^ {k} = 1 ^ {k}\right) \right\} \\ = 1 - p ^ {k} - (1 - p) ^ {k} + \max  _ {\pi} \left\{\pi \left(Y = 0, X ^ {k} = 0 ^ {k}\right) + \pi \left(Y = 1, X ^ {k} = 1 ^ {k}\right) \right\}. \\ \end{array}
$$
Consider the transport plan  $\pi$  given by  $\pi(1^k, 1) = \min(p^k, q)$ ,  $\pi(1^k, 0) = p^k - \min(p^k, q)$ ,  $\pi(0^k, 0) = \min((1 - p)^k, 1 - q)$ , and  $\pi(0^k, 1) = (1 - p)^k - \min((1 - p)^k, 1 - q)$ . It can be checked that this is a valid transport plan. To see this matches the upper bound on the optimal cost from Lemma 3, notice that
$$
\begin{array}{l} 1 - p ^ {k} - (1 - p) ^ {k} + \max  _ {\pi} \left\{\pi \left(Y = 0, X ^ {k} = 0 ^ {k}\right) + \pi \left(Y = 1, X ^ {k} = 1 ^ {k}\right) \right\} \\ = 1 - p ^ {k} - (1 - p) ^ {k} + \min  (p ^ {k}, q) + \min  ((1 - p) ^ {k}, 1 - q). \\ \end{array}
$$
If  $p^k \leq q$  and  $(1 - p)^k \leq 1 - q$ , then the above equation simplifies to 1 and (10) also simplifies to 1. If  $p^k > q$  and  $(1 - p)^k \leq 1 - q$ , then the above equation simplifies to  $1 + q - p^k$  and (10) also simplifies to the same quantity. Similarly, the proof applies for  $p^k \leq q$  and  $(1 - p)^k > 1 - q$ .
Figure 5:  $p = U(d)$  and  $q = U(d / r)$ . The optimal acceptance probability is
$$
\alpha_ {k} (U (d), U (d / r)) = 1 - (1 - 1 / r) ^ {k}.
$$
We first prove  $\alpha^k (U(d), U(d / r)) \geq 1 - (1 - 1 / r)^k$  by a construction. Let  $S(X^k)$  be the set of unique symbols in  $X^k$ . Consider the following transport plan, where  $Y$  is drawn uniformly from  $S(\bar{X}^k) \cap [d / r]$  and draws a new uniform sample from  $[d / r]$  if  $S(X^k) \cap [d / r] = \emptyset$ . Observe that since  $U(d)$  is uniform over  $[d]$ , this is a valid transport plan and furthermore,
$$
\alpha_ {k} (U (d), U (d / r)) \geq \Pr (S (X) ^ {k} \cap [ d / r ] \neq \emptyset) = 1 - (1 - 1 / r) ^ {k}.
$$
The upper bound follows by setting  $\Omega_0 = [d] \setminus [d / r]$  in Lemma 3.
$$
\alpha_ {k} (U (d), U (d / r)) \leq \Pr (S (X ^ {k}) \cap [ d / r ] \neq \emptyset) = 1 - (1 - 1 / r) ^ {k}.
$$
B.2 Acceptance probability of K-SEQ for the example in Figure 5
In this section, we show that for the example in Figure 5,  $\kappa$ -SEQ achieves the optimal acceptance accuracy. In this case,  $p = U(d)$  and  $q = U(d / r)$ . Recall that the optimal acceptance probability is
For  $U(d)$  and  $U(d / r)$ , we have
$$
\beta (\rho) = \sum_ {x \in [ d ]} \min  \{p (x), q (x) / \rho \} = \frac {1}{\max  \{r , \rho \}}.
$$
And hence solving  $1 - (1 - \beta (\rho))^{k} = \rho \beta (\rho)$  gives  $\rho^{*} = r(1 - (1 - 1 / r)^{k})$ . And be Theorem 1, we have
$$
\alpha \left(\pi_ {\rho^ {*}} ^ {\mathrm {K} - \mathrm {S E Q}}\right) \geq 1 - \left(1 - \beta \left(\rho^ {*}\right)\right) ^ {k} = 1 - \left(1 - 1 / r\right) ^ {k}.
$$
And the equality holds since this is an upper bound for any coupling.
B.3 Comparison to multi-round rejection sampling in [20, 29]
In this section, we compare our proposed draft selection algorithms (OTM and K-SEQ) to the multi-round rejection sampling algorithm (MULTI-ROUND) in concurrent and recent work of [20, 29] (see Algorithm 1 in [29]) using the example of Bernoulli distributions. As Figure 6 demonstrates, both our proposed algorithms outperform their algorithm. The advantage of OTM is demonstrated by the fact it is the optimal algorithm under the validity guarantee of the final accepted token. Our proposed efficient algorithm K-SEQ also outperforms MULTI-ROUND for the considered examples. We leave a systematic comparison of the algorithms as future work.
Figure 6: Acceptance probability comparison among OTM- $k$ , K-SEQ and MULTI-ROUND when  $p = \operatorname{Ber}(0.25)$  and  $q = \operatorname{Ber}(b)$ . When  $b = 0.25$ , all three algorithms achieve an acceptance probability of 1, and hence omitted in the plot.
C Analysis of SpecTr
We start by proving the following lemma on  $\rho^{*}$ .
Lemma 4. Let
$$
f (\rho) = 1 - \left(1 - \beta_ {p, q} (\rho)\right) ^ {k} - \rho \beta_ {p, q} (\rho).
$$
Then we have Let  $\rho^{*}$  be the solution to Eq. (6). Then when  $d_{\mathrm{TV}}(p,q)\in (0,1)$
$f(\rho)$  is monotone in  $\rho$  in  $[1,\infty)$
-  $\rho^{*} \in \left[1, \min \left\{k, \max_{x} \frac{q(x)}{p(x)}\right\}\right]$ .
Proof. It would enough to prove the followings: (1)  $f(\rho)$  is monotone in  $\rho$  in  $[1, \infty)$ ; (2)  $f(1) \geq 0$ ; (3)  $f(k) \leq 0$ ; (4)  $f\left(\max_x \frac{q(x)}{p(x)}\right) \leq 0$ .
To see (1), since  $\beta_{p,q}(\rho)$  is decreasing in  $\rho$ , so is  $1 - (1 - \beta_{p,q}(\rho))^k$ . Moreover,  $\rho \beta_{p,q}(\rho) = \sum_x \min \{\rho p(x), q(x)\}$ , which is non-decreasing in  $\rho$ . Hence we have  $1 - (1 - \beta_{p,q}(\rho))^k - \rho \beta_{p,q}(\rho)$  is decreasing.
To see (2), note that when  $\rho = 1$ ,  $\beta_{p,q}(\rho) = 1 - d_{\mathrm{TV}}(p,q)$ . Hence we have
$$
1 - \big (1 - \beta_ {p, q} (1) \big) ^ {k} = 1 - d _ {\mathrm {T V}} (p, q) ^ {k} \geq 1 - d _ {\mathrm {T V}} (p, q).
$$
When  $\rho = k$ , (3) holds since for  $x \in [0,1]$ , we have  $1 - (1 - x)^k \leq kx$ . Moreover, when  $\rho = \max_x \frac{q(x)}{p(x)} > 1$ , we have  $\beta_{p,q}(\rho) = 1 / \rho$  and (4) holds since
$$
1 - (1 - \beta_ {p, q} (\rho)) ^ {k} = 1 - (1 - 1 / \rho) ^ {k} <   1 = \rho \cdot 1 / \rho .
$$
Next we prove Theorem 1, we will break the proof into four parts: (1) computation efficiency; (2)  $\pi_{\rho}^{\mathrm{K - SEQ}}$  is a valid transport plan; (3) acceptance probability; (4) optimality guarantee of  $\pi_{\rho^{*}}^{\mathrm{K - SEQ}}$ .
Computation efficiency. Note that Lemma 4 immediately implies that  $\rho^{*}$  can be computed up to arbitrary accuracy  $\delta$  in time  $O(|\Omega| \log((k - 1)/\delta))$  using binary search over  $[1, k]$ .
Valid transport plan. We next prove that  $\pi_{\rho}^{\mathrm{K - SEQ}}$  is a valid transport plan when  $\rho \geq \rho^{*}$ . By Lemma 4, when  $\rho \geq \rho^{*}$ , we have  $1 - (1 - \beta_{p,q}(\rho))^k \geq \rho \beta_{p,q}(\rho)$ . Recall that  $p_{\mathrm{acc}} = 1 - (1 - \beta_{p,q}(\rho))^k$ , and
$$
\forall x \in \Omega , p ^ {\mathrm {r e s}} (x) = \frac {q (x) - \min \left\{p (x) , \frac {q (x)}{\rho} \right\} \frac {p _ {\mathrm {a c c}}}{\beta_ {p , q} (\rho)}}{1 - p _ {\mathrm {a c c}}}.
$$
$\forall x \in \Omega$ , we have
$$
\min \left\{p (x), \frac {q (x)}{\rho} \right\} \frac {p _ {\mathrm {a c c}}}{\beta_ {p , q} (\rho)} \leq \frac {1 - (1 - \beta_ {p , q} (\rho)) ^ {k}}{\rho \beta_ {p , q} (\rho)} q (x) \leq q (x),
$$
this implies  $p^{\mathrm{res}}(x)\geq 0$  for all  $x\in \Omega$ . Moreover,
$$
\sum_ {x \in \Omega} p ^ {\mathrm {r e s}} (x) = \sum_ {x \in \Omega} \frac {q (x) - \min \left\{p (x) , \frac {q (x)}{\rho} \right\} \frac {p _ {\mathrm {a c c}}}{\beta_ {p , q} (\rho)}}{1 - p _ {\mathrm {a c c}}} = 1.
$$
Hence  $p^{\mathrm{res}}$  is a valid distribution. It remains to show that the marginal of  $Y$  is  $q$ . We first compute the probability of the output  $Y = x$ . Note that probability that  $Y = X_{1}$  is
$$
p (X _ {1}) \min  \left(1, \frac {q (X _ {1})}{\rho p (X _ {1})}\right) = \min  \left(p (X _ {1}), \frac {q (X _ {1})}{\rho}\right).
$$
$$
\operatorname * {P r} (Y = X _ {1} = x) = \min \left(p (x), \frac {q (x)}{\rho}\right).
$$
$$
\operatorname * {P r} (Y = X _ {1}) = \sum_ {x} \min \left(p (x), \frac {q (x)}{\rho}\right) = \beta (\rho).
$$
Similarly, probability that
$$
\Pr (Y = X _ {2} = x) = \Pr (Y \neq X _ {1}) \Pr (Y = X _ {2} | Y \neq X _ {1}) = (1 - \beta_ {p, q} (\rho)) \min  \left(p (x), \frac {q (x)}{\rho}\right).
$$
$$
\begin{array}{l} \Pr (Y = x, \text {o n e o f} X ^ {k} \text {i s a c c e p t e d}) \\ = \sum_ {i = 0} ^ {k - 1} \Pr \left(X _ {1}, \dots , X _ {i} \text {a r e r e j e c t e d}, X _ {i + 1} \text {i s a c c e p t e d , a n d} X _ {i + 1} = x\right) \\ = \sum_ {i = 0} ^ {k - 1} (1 - \beta_ {p, q} (\rho)) ^ {i} \cdot p (x) \cdot \min  \left\{1, \frac {q (x)}{p (x) \rho} \right\} \\ = \min  \left\{p (x), \frac {q (x)}{\rho} \right\} \cdot \sum_ {i = 0} ^ {k - 1} \left(1 - \beta_ {p, q} (\rho)\right) ^ {i} \\ = \min \left\{p (x), \frac {q (x)}{\rho} \right\} \frac {1 - (1 - \beta_ {p , q} (\rho)) ^ {k}}{\beta_ {p , q} (\rho)} \\ \end{array}
$$
Summing over all symbols  $x$  yields
$$
\Pr (\text {o n e o f} X ^ {k} \text {i s a c c e p t e d}) = \sum_ {x} \min  \left\{p (x), \frac {q (x)}{\rho} \right\} \frac {1 - \left(1 - \beta_ {p , q} (\rho)\right) ^ {k}}{\beta_ {p , q} (\rho)} = 1 - \left(1 - \beta_ {p, q} (\rho)\right) ^ {k}.
$$
$$
\begin{array}{l} \Pr (Y = x) = \Pr (Y = x, \text {o n e o f} X ^ {k} \text {i s a c c e p t e d}) + (1 - \Pr (\text {o n e o f} X ^ {k} \text {i s a c c e p t e d})) p ^ {\text {r e s}} (x) \\ = \min \left\{p (x), \frac {q (x)}{\rho} \right\} \frac {1 - (1 - \beta_ {p , q} (\rho)) ^ {k}}{\beta_ {p , q} (\rho)} \\ + \left(1 - (1 - \beta_ {p, q} (\rho)) ^ {k}\right) \frac {q (x) - \min \left\{p (x) , \frac {q (x)}{\rho} \right\} \frac {1 - (1 - \beta_ {p , q} (\rho)) ^ {k}}{\beta_ {p , q} (\rho)}}{1 - (1 - \beta_ {p , q} (\rho)) ^ {k}} \\ = q (x). \\ \end{array}
$$
Acceptance probability. The acceptance probability holds since
$$
\alpha \left(\pi_ {\rho} ^ {\mathrm {K - S E Q}}\right) \geq \Pr (\text {o n e o f} X ^ {k} \text {i s a c c e p t e d}) = 1 - \left(1 - \beta_ {p, q} (\rho)\right) ^ {k}.
$$
Optimality guarantee of  $\pi_{\rho^*}^{\mathbf{K - SEQ}}$ . It can be seen that  $\beta (\rho)$  is decreasing in  $\rho$ , and so is  $1 - (1 - \beta_{p,q}(\rho))^k$ . Hence we have
$$
\alpha (\pi_ {\rho^ {*}} ^ {\mathrm {K - S E Q}}) \geq 1 - (1 - \beta_ {p, q} (\rho^ {*})) ^ {k} \geq 1 - (1 - \beta_ {p, q} (k)) ^ {k} = c _ {k} (p, q) \cdot \sum_ {x} \min \{k p (x), q (x) \},
$$
$$
c _ {k} (p, q) = \frac {1 - (1 - \beta_ {p , q} (k)) ^ {k}}{k \beta_ {p , q} (k)} \in [ 1 - (1 - 1 / k) ^ {k}, 1).
$$
The statement holds since  $f(x) = \frac{1 - (1 - x)^k}{kx}$  in monotonically decreasing when  $x \in (0, 1/k]$  and  $f(1/k) = 1 - (1 - 1/k)^k$ ,  $\lim_{x \to 0^+} f(x) = 1$ .
Moreover,  $\forall x\geq 0,kx\geq 1 - (1 - x)^{k}$  . Hence we have
$$
\begin{array}{l} \alpha \left(\pi_ {\rho^ {*}} ^ {\mathrm {K} - \mathrm {S E Q}}\right) \geq \left(1 - (1 - 1 / k) ^ {k}\right) \cdot \sum_ {x} \min  \{k p (x), q (x) \} \\ \geq \left(1 - (1 - 1 / k) ^ {k}\right) \sum_ {x} \min  \{1 - (1 - p (x)) ^ {k}, q (x) \} (11) \\ \geq \left(1 - (1 - 1 / k) ^ {k}\right) \alpha_ {k} (p, q), n o n u m b e r (12) \\ \end{array}
$$
where the last inequality is due to the upper bound in Lemma 3 with  $\Omega_0 = \Omega$ .
We prove the theorem via induction. When  $L = 1, \tau \in \{1,2\}$ . Let  $k = |S|$ . Since for the first step,  $f_{\pi}$  in Algorithm 3 is a valid transport plan from  $\mathcal{M}_s(\cdot \mid x^t)^{\otimes k}$  to  $\mathcal{M}_b(\cdot \mid x^t)$ . We have  $Y_1 \sim \mathcal{M}_b(\cdot \mid x^t)$ , which completes the proof when  $\tau = 1$ . When  $\tau = 2$ , we have  $Y_2 \sim \mathcal{M}_b(\cdot \mid x^t, Y_1)$  as stated in Step 5 of Algorithm 3. Hence the statement holds.
Suppose the theorem holds for  $L = \ell \geq 1$ , we next prove that it holds for  $L = \ell + 1$ . Let  $Y_{x^t}^\tau$  be the output sequence given context  $x^t$ . When  $\tau = 1$ , since for the first step,  $f_{\pi}$  in Algorithm 3 is a valid transport plan from  $\mathcal{M}_s(\cdot \mid x^t)^{\otimes k}$  to  $\mathcal{M}_b(\cdot \mid x^t)$ , we have  $Y_1 \sim \mathcal{M}_b(\cdot \mid x^t)$ . When  $\tau > 1$ ,  $S_{\mathrm{next}} \neq \emptyset$  and by the assumption in Eq. (8),  $S_{\mathrm{next}}$  contains  $k' = |S_{\mathrm{next}}|$  drafts from  $\mathcal{M}_s(\cdot \mid (x^t, Y_1))$  with length  $\ell$ . Let  $Y_{x^t, Y_1}^{\tau'}$  be the output sequence given context  $(x^t, Y_1)$ , by the induction assumption, we have for any  $\tau_0 \in [1, \ell + 1]$ , and any  $\tau_0$ -length, sequence  $o^{\tau_0} = (o(1), \dots, o(\tau_0)) \in \Omega^{\tau_0}$ , we have
$$
\operatorname * {P r} \left(Y _ {x ^ {t}, Y _ {1}} ^ {\tau^ {\prime}} = o ^ {\tau_ {0}} | \tau^ {\prime} = \tau_ {0}\right) = \Pi_ {i = 1} ^ {\tau_ {0}} \mathcal {M} _ {b} (o (i) \mid x ^ {t}, Y _ {1}, o ^ {i - 1}).
$$
Note that in this case  $\tau = \tau' + 1$ , and for any  $(\tau_0 + 1)$ -length sequence  $o^{\tau_0 + 1} = (o(1), \ldots, o(\tau_0), o(\tau_0 + 1)) \in \Omega^{\tau_0 + 1}$ , we have
$$
\begin{array}{l} \Pr \left(Y _ {x ^ {t}} ^ {\tau} = o ^ {\tau_ {0} + 1} | \tau = \tau_ {0} + 1\right) = \Pr \left(Y _ {1} = o _ {1}\right) \cdot \Pr \left(Y _ {x ^ {t}, o _ {1}} ^ {\tau^ {\prime}} = o ^ {\tau_ {0}} | \tau^ {\prime} = \tau_ {0}\right) \\ = \mathcal {M} _ {b} (o (1) \mid x ^ {t}) \cdot \Pi_ {i = 1} ^ {\tau_ {0}} \mathcal {M} _ {b} (o (i + 1) \mid x ^ {t}, o ^ {i}) \\ = \Pi_ {i = 1} ^ {\tau_ {0} + 1} \mathcal {M} _ {b} (o (i) \mid x ^ {t}, o ^ {i - 1}). \\ \end{array}
$$
Combining the two cases, we complete the proof.
D Candidate set construction via a prefix-tree
As discussed in Section 1, the size of the draft set  $S$  is constrained by the number of parallel computations that can be supported in the hardware. Hence it is important to design the draft set carefully to allow for a longer sequence of accepted candidate sets. In addition to the i.i.d. draft set selection approach listed in Section 7, we present an algorithm that samples a draft set that forms the leaves of a prefix tree. Given a draft set size  $K$ , the algorithm can be specified by a sequence of parameter  $(k_{1}, k_{2}, \ldots, k_{L})$  satisfying  $\prod_{i=1}^{L} k_{i} = K$ .
The algorithm starts with a root node with sequence  $x^{1:t}$  and forms a prefix tree of depth  $L$ . At depth  $i \in [1:L - 1]$ , each node is expanded by a factor of  $k_{i + 1}$  and each of its children will contain a sequence that satisfies: (1) Its prefix agrees with the sequence in the parent node; (2) The next token is sampled from the conditional probability given the prefix in small model. These child nodes will be at depth  $i + 1$  and the process goes until it hits depth  $L$ . We give a detailed description of the algorithm in Algorithm 4.
Algorithm 4 Draft set selection via prefix-tree.
Input: Input sequence  $x^t$ ; expansion factors at each level:  $(k_1, k_2, \ldots, k_L)$ .  
1:  $S_0 = \{x^t\}$ .  
2: for  $i = 0, 1, 2, \ldots, L - 1$  do  
3:  $S_{i + 1} = \emptyset$ .  
4: for all seq  $\in S_i$  do  
5: Sample  $k_{i + 1}$  i.i.d. tokens  $X_1, X_2, \ldots, X_{k_{i + 1}}$  from  $\mathcal{M}_b(\cdot | \text{seq})$ .  
6:  $S_{i + 1} = S_{i + 1} \cup \{(\text{seq}, X_i), i = 1, 2, \ldots, k_{i + 1}\}$ .  
7: end for  
8: end for  
9: Return  $S_L$ .
In this section, we perform a detailed investigation of different factors that affect the speed of SpecTr with smaller transformer models. We train decoder-only transformer models on the LM1B
Table 2: Average latency with parallelization along the time axis and batch axis. We report average latency with standard deviation over 1,000 runs using a  $97M$  transformer relative to length  $= 4$  and batch  $= 1$  on GPU.
Relative latency	batch = 1	batch = 2	batch = 4	batch = 8
length = 4	1.00 ± 0.16	1.01 ± 0.15	1.06 ± 0.10	1.10 ± 0.16
length = 8	1.01 ± 0.18	1.09 ± 0.25	1.10 ± 0.09	1.42 ± 0.4
Table 3: Average latency with parallelization along the time axis and batch axis. We report average latency with standard deviation over 1,000 runs using a  $6M$  transformer relative to the  $97M$  transformer with length  $= 4$  and batch  $= 1$  on GPU.
Relative latency	batch = 1	batch = 2	batch = 4	batch = 8
length = 4	0.18 ± 0.02	0.19 ± 0.04	0.18 ± 0.09	0.20 ± 0.13
length = 8	0.17 ± 0.04	0.19 ± 0.05	0.16 ± 0.02	0.18 ± 0.04
dataset based on the example provided in the FLAX library [15]. For the draft model, we use transformer models with  $2M$ ,  $6M$  and  $20M$  parameters, and for the large model we use a  $97M$  parameter transformer model.
We first provide a verification of the computational model introduced in Section 1 by reporting the latencies of using the large model to compute the probabilistic distributions with parallelization over time and batch axes. As shown in Table 2, the latency stays roughly constant in these settings.