AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Authors: name: Tong Wu; name: Zhihao Fan; name: Xiao Liu; name: Yeyun Gong; name: Yelong Shen; name: Jian Jiao; name: Hai-Tao Zheng; name: Juntao Li; name: Zhongyu Wei; name: Jian Guo; name: Nan Duan; name: Weizhu Chen
Published: 2023-05-16 15:10:22

Abstract: Diffusion models have gained significant attention in the realm of image
generation due to their exceptional performance. Their success has been
recently expanded to text generation via generating all tokens within a
sequence concurrently. However, natural language exhibits a far more pronounced
sequential dependency in comparison to images, and the majority of existing
language models are trained utilizing a left-to-right auto-regressive approach.
To account for the inherent sequential characteristic of natural language, we
introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that
the generation of tokens on the right depends on the generated ones on the
left, a mechanism achieved through employing a dynamic number of denoising
steps that vary based on token position. This results in tokens on the left
undergoing fewer denoising steps than those on the right, thereby enabling them
to generate earlier and subsequently influence the generation of tokens on the
right. In a series of experiments on various text generation tasks including
text summarization, machine translation, and common sense generation,
AR-Diffusion clearly demonstrated the superiority over existing diffusion
language models and that it can be 100timessim600times faster when
achieving comparable results. Our code will be publicly released.