TESS: Text-to-Text Self-Conditioned Simplex Diffusion
Authors: name: Rabeeh Karimi Mahabadi; name: Jaesung Tae; name: Hamish Ivison; name: James Henderson; name: Iz Beltagy; name: Matthew E. Peters; name: Arman Cohan
Published: 2023-05-15 06:33:45

Abstract: Diffusion models have emerged as a powerful paradigm for generation,
obtaining strong performance in various domains with continuous-valued inputs.
Despite the promises of fully non-autoregressive text generation, applying
diffusion models to natural language remains challenging due to its discrete
nature. In this work, we propose Text-to-text Self-conditioned Simplex
Diffusion (TESS), a text diffusion model that is fully non-autoregressive,
employs a new form of self-conditioning, and applies the diffusion process on
the logit simplex space rather than the typical learned embedding space.
Through extensive experiments on natural language understanding and generation
tasks including summarization, text simplification, paraphrase generation, and
question generation, we demonstrate that TESS outperforms state-of-the-art
non-autoregressive models and is competitive with pretrained autoregressive
sequence-to-sequence models.