SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Similar to Table 2, we report relative latency when parallelizing across the time and batch axes using the small  $6M$  draft model in Table 3. In Table 3, the reported relative latencies are relative to the large  $97M$  model to get a sense of the relative cost of sampling multiple drafts with the small model compared to the large model.
To see how the size of size of the draft model will affect the block efficiency, we also include results for varying draft model sizes with the same  $97M$  large model for LM1B in Table 4. These draft models were produced by either halving  $(2M)$  or doubling  $(20M)$  the original  $6M$  draft model's number of layers, embedding dimension, MLP dimension, and number of attention heads. As expected, the larger draft models improve all speculative methods' block efficiency with SpecTr maintaining the best performance across all draft model sizes.
Table 4: Experimental results on the LM1B dataset with varying draft model sizes and the  $97M$  transformer as the large model. All results are over 1000 test prompts averaged over three different random seeds and sampling temperature of 1.0 for both the draft and large models.
Draft model	Algorithm	K	L	Block efficiency
2M Transformer	Baseline	-	-	1.00
Speculative	1	4	1.86 ± 0.02
SpecTr	2	4	2.07 ± 0.01
SpecTr	4	4	2.32 ± 0.00
SpecTr	8	4	2.56 ± 0.01
Speculative	1	8	1.91 ± 0.01
SpecTr	2	8	2.15 ± 0.01
SpecTr	4	8	2.41 ± 0.00
SpecTr	8	8	2.68 ± 0.01
6M Transformer	Baseline	-	-	1.00
Speculative	1	4	2.21 ± 0.01
SpecTr	2	4	2.43 ± 0.01
SpecTr	4	4	2.74 ± 0.01
SpecTr	8	4	2.99 ± 0.02
Speculative	1	8	2.33 ± 0.01
SpecTr	2	8	2.61 ± 0.02
SpecTr	4	8	2.96 ± 0.03
SpecTr	8	8	3.27 ± 0.02
20M Transformer	Baseline	-	-	1.00
Speculative	1	4	2.71 ± 0.01
SpecTr	2	4	2.96 ± 0.00
SpecTr	4	4	3.28 ± 0.02
SpecTr	8	4	3.49 ± 0.03
Speculative	1	8	3.12 ± 0.02
SpecTr	2	8	3.48 ± 0.04
SpecTr	4	8	3.85 ± 0.05
SpecTr	8	8	4.15 ± 0.04