Title: Training-Free Multi-Token Prediction via Probing
Primary area: foundation or frontier models, including LLMs
Keywords: Multi-token prediction, training-free inference, Inference acceleration, SpecBench

Abstract: Large Language Models (LLMs) possess latent capabilities for multi-token prediction (MTP), despite being primarily trained for next-token generation. We introduce a simple, **novel, and training-free MTP** method that probes an LLM using **on-the-fly generated mask tokens** derived from the model’s own embedding space. These mask tokens guide the model to predict multiple future tokens in parallel without modifying model weights or relying on external draft models.
To assess the quality of these predictions, we propose a dynamic draft tree construction mechanism based on cumulative token probabilities, coupled with a simple pruning algorithm that removes redundant token paths. Our method supports multiple mask token designs, enabling flexible probing strategies to improve parallel prediction quality.
We also define block complexity—the number of tokens processed per model iteration—as a key setting for controlling compute usage during inference. Under equal block complexity, our approach consistently outperforms existing training-free baselines such as Lookahead decoding and Prompt-lookup-decoding in terms of **block efficiency** (acceptance rate), significantly **reducing number of model calls**.
We evaluate our method on Spec-Bench using `LLaMA3` and `Qwen3` models, providing both quantitative and qualitative analyses. Our probing-based MTP improves block efficiency by $\sim12\\%$ for `LLaMA3` and $8–12\\%$ for `Qwen3` models over baselines **without retraining or auxiliary models**.