Ideas in Inference-time Scaling can Benefit Generative Pre-training
  Algorithms
Authors: name: Jiaming Song, name: Linqi Zhou

Recent years have seen significant advancements in foundation models through
generative pre-training, yet algorithmic innovation in this space has largely
stagnated around autoregressive models for discrete signals and diffusion
models for continuous signals. This stagnation creates a bottleneck that
prevents us from fully unlocking the potential of rich multi-modal data, which
in turn limits the progress on multimodal intelligence. We argue that an
inference-first perspective, which prioritizes scaling efficiency during
inference time across sequence length and refinement steps, can inspire novel
generative pre-training algorithms. Using Inductive Moment Matching (IMM) as a
concrete example, we demonstrate how addressing limitations in diffusion
models' inference process through targeted modifications yields a stable,
single-stage algorithm that achieves superior sample quality with over an order
of magnitude greater inference efficiency.