Flowing from Words to Pixels: A Framework for Cross-Modality Evolution
Authors: name: Qihao Liu; name: Xi Yin; name: Alan Yuille; name: Andrew Brown; name: Mannat Singh
Published: 2024-12-19 18:59:56

Abstract: Diffusion models, and their generalization, flow matching, have had a
remarkable impact on the field of media generation. Here, the conventional
approach is to learn the complex mapping from a simple source distribution of
Gaussian noise to the target media distribution. For cross-modal tasks such as
text-to-image generation, this same mapping from noise to image is learnt
whilst including a conditioning mechanism in the model. One key and thus far
relatively unexplored feature of flow matching is that, unlike Diffusion
models, they are not constrained for the source distribution to be noise.
Hence, in this paper, we propose a paradigm shift, and ask the question of
whether we can instead train flow matching models to learn a direct mapping
from the distribution of one modality to the distribution of another, thus
obviating the need for both the noise distribution and conditioning mechanism.
We present a general and simple framework, CrossFlow, for cross-modal flow
matching. We show the importance of applying Variational Encoders to the input
data, and introduce a method to enable Classifier-free guidance. Surprisingly,
for text-to-image, CrossFlow with a vanilla transformer without cross attention
slightly outperforms standard flow matching, and we show that it scales better
with training steps and model size, while also allowing for interesting latent
arithmetic which results in semantically meaningful edits in the output space.
To demonstrate the generalizability of our approach, we also show that
CrossFlow is on par with or outperforms the state-of-the-art for various
cross-modal / intra-modal mapping tasks, viz. image captioning, depth
estimation, and image super-resolution. We hope this paper contributes to
accelerating progress in cross-modal media generation.