Mirror Speculative Decoding: Breaking the Serial Barrier in LLM
  Inference
Authors: name: Nikhil Bhendawade, name: Kumari Nishu, name: Arnav Kundu, name: Chris Bartels, name: Minsik Cho, name: Irina Belousova
Organization: Apple

Speculative decoding accelerates LLM inference by using a draft model to look
ahead, but gains are capped by the cost of autoregressive draft generation:
increasing draft size elevates acceptance rates but introduces additional
latency overhead exacerbating the speed-accuracy tradeoff. Prior methods
(Medusa, Hydra, EAGLE) partially reduce draft cost but either degrade
acceptance or introduce overheads that limit scaling. We present Mirror
Speculative Decoding (Mirror-SD), an inference algorithm that breaks the
latency-acceptance tradeoff. Mirror-SD launches branch-complete rollouts from
early-exit signals in parallel with the target model's suffix and explicitly
maps computation across heterogeneous accelerators (GPU and NPU) to exploit
cross-device parallelism. The draft speculates forward continuations for the
target to verify, while the target simultaneously speculates correction paths
for the draft, converting speculation into two complementary execution
pipelines. To further cut draft latency without weakening acceptance semantics,
we add speculative streaming so the draft emits multiple tokens per step. This
dual strategy of parallel heterogeneous execution plus multi-token speculative
streaming pushes speculative decoding toward its ideal regime of high
acceptance with low overhead. On SpecBench with server-scale models from 14B to
66B parameters, Mirror-SD delivers consistent end-to-end gains, achieving
2.8x-5.8x wall-time speedups across diverse tasks and a 30% average relative
improvement over the strongest baseline, EAGLE3.