Inconsistencies In Consistency Models: Better ODE Solving Does Not Imply
  Better Samples
Authors: name: Noël Vouitsis, name: Rasa Hosseinzadeh, name: Brendan Leigh Ross, name: Valentin Villecroze, name: Satya Krishna Gorti, name: Jesse C. Cresswell, name: Gabriel Loaiza-Ganem
Organization: Layer 6 AI

Although diffusion models can generate remarkably high-quality samples, they
are intrinsically bottlenecked by their expensive iterative sampling procedure.
Consistency models (CMs) have recently emerged as a promising diffusion model
distillation method, reducing the cost of sampling by generating high-fidelity
samples in just a few iterations. Consistency model distillation aims to solve
the probability flow ordinary differential equation (ODE) defined by an
existing diffusion model. CMs are not directly trained to minimize error
against an ODE solver, rather they use a more computationally tractable
objective. As a way to study how effectively CMs solve the probability flow
ODE, and the effect that any induced error has on the quality of generated
samples, we introduce Direct CMs, which directly minimize this error.
Intriguingly, we find that Direct CMs reduce the ODE solving error compared to
CMs but also result in significantly worse sample quality, calling into
question why exactly CMs work well in the first place. Full code is available
at: https://github.com/layer6ai-labs/direct-cms.