Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD
Authors: name: Emiel Hoogeboom, name: David Ruhe, name: Jonathan Heek, name: Thomas Mensink, name: Tim Salimans
Organization: Deepmind

It is currently difficult to distill discrete diffusion models. In contrast, continuous diffusion literature has many distillation approaches methods that can reduce sampling steps to a handful.
  Our method, Discrete Moment Matching Distillation (D-MMD), leverages ideas that have been highly successful in the continuous domain. Whereas previous discrete distillation methods collapse, D-MMD maintains high quality and diversity (given sufficient sampling steps). This is demonstrated on both text and image datasets. Moreover, the newly distilled generators can outperform their teachers.